[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 491907750 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2528MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003395] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.005015] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008032] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009017] pid_max: default: 32768 minimum: 301 [ 0.010153] LSM: Security Framework initializing [ 0.011069] Yama: becoming mindful. [ 0.012051] SELinux: Initializing. [ 0.013083] *** VALIDATE selinux *** [ 0.021731] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025892] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027179] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028119] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029128] *** VALIDATE tmpfs *** [ 0.031271] *** VALIDATE proc *** [ 0.032272] *** VALIDATE cgroup *** [ 0.033013] *** VALIDATE cgroup2 *** [ 0.034311] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035156] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036012] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037033] Spectre V2 : User space: Vulnerable [ 0.038011] Speculative Store Bypass: Vulnerable [ 0.040768] debug: unmapping init [mem 0xffffffff8a659000-0xffffffff8a660fff] [ 0.042904] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043693] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044025] ... version: 2 [ 0.045017] ... bit width: 48 [ 0.046014] ... generic registers: 4 [ 0.047014] ... value mask: 0000ffffffffffff [ 0.048017] ... max period: 00007fffffffffff [ 0.049015] ... fixed-purpose events: 3 [ 0.050011] ... event mask: 000000070000000f [ 0.051379] rcu: Hierarchical SRCU implementation. [ 0.053503] smp: Bringing up secondary CPUs ... [ 0.054640] x86: Booting SMP configuration: [ 0.055029] .... node #0, CPUs: #1 #2 #3 [ 0.059552] smp: Brought up 1 node, 4 CPUs [ 0.061015] smpboot: Max logical packages: 1 [ 0.062025] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.130022] node 0 deferred pages initialised in 65ms [ 0.133043] devtmpfs: initialized [ 0.134000] x86/mm: Memory block size: 128MB [ 0.135948] gcov: version magic: 0x41383552 [ 0.138342] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.139082] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.140302] pinctrl core: initialized pinctrl subsystem [ 0.141261] [ 0.142011] ************************************************************* [ 0.143016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.144014] ** ** [ 0.145014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.146016] ** ** [ 0.147015] ** This means that this kernel is built to expose internal ** [ 0.148014] ** IOMMU data structures, which may compromise security on ** [ 0.149015] ** your system. ** [ 0.150016] ** ** [ 0.151066] ** If you see this message and you are not debugging the ** [ 0.152015] ** kernel, report this immediately to your vendor! ** [ 0.153016] ** ** [ 0.154016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.155014] ************************************************************* [ 0.156773] NET: Registered protocol family 16 [ 0.157439] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.158061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.159072] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.161127] cpuidle: using governor menu [ 0.163659] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.168584] PCI: Using configuration type 1 for base access [ 0.170143] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.180059] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.181044] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.182270] cryptd: max_cpu_qlen set to 1000 [ 0.184313] ACPI: Added _OSI(Module Device) [ 0.185018] ACPI: Added _OSI(Processor Device) [ 0.186015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.187015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.191303] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.193586] ACPI: Interpreter enabled [ 0.194069] ACPI: PM: (supports S0 S3 S4 S5) [ 0.195014] ACPI: Using IOAPIC for interrupt routing [ 0.196106] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.197555] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.208066] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.209037] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.210032] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.211082] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.212889] acpiphp: Slot [2] registered [ 0.213000] acpiphp: Slot [5] registered [ 0.216130] acpiphp: Slot [6] registered [ 0.217136] acpiphp: Slot [7] registered [ 0.218114] acpiphp: Slot [8] registered [ 0.220200] acpiphp: Slot [9] registered [ 0.222132] acpiphp: Slot [10] registered [ 0.224139] acpiphp: Slot [3] registered [ 0.226165] acpiphp: Slot [4] registered [ 0.227107] acpiphp: Slot [11] registered [ 0.229259] acpiphp: Slot [12] registered [ 0.231177] acpiphp: Slot [13] registered [ 0.233187] acpiphp: Slot [14] registered [ 0.235180] acpiphp: Slot [15] registered [ 0.237132] acpiphp: Slot [16] registered [ 0.239164] acpiphp: Slot [17] registered [ 0.240195] acpiphp: Slot [18] registered [ 0.242199] acpiphp: Slot [19] registered [ 0.244147] acpiphp: Slot [20] registered [ 0.246156] acpiphp: Slot [21] registered [ 0.247144] acpiphp: Slot [22] registered [ 0.248157] acpiphp: Slot [23] registered [ 0.249202] acpiphp: Slot [24] registered [ 0.251150] acpiphp: Slot [25] registered [ 0.252177] acpiphp: Slot [26] registered [ 0.253131] acpiphp: Slot [27] registered [ 0.255126] acpiphp: Slot [28] registered [ 0.256116] acpiphp: Slot [29] registered [ 0.258108] acpiphp: Slot [30] registered [ 0.259095] acpiphp: Slot [31] registered [ 0.261088] PCI host bridge to bus 0000:00 [ 0.262021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.265024] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.266021] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.269030] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.273033] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.276027] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.277226] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.281200] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.285020] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.296878] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.301062] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.304028] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.307028] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.311030] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.314725] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.317920] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.321101] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.324726] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.330019] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.343017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.349019] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.355045] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.364021] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.374019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.394020] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.404680] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.415029] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.429026] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.451028] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.468734] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.476018] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.484024] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.498000] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.508624] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.517016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.533018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.560016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.573314] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.582019] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.590016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.607021] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.616629] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.622016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.630017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.648021] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.659108] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.661446] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.663352] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.666434] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.668212] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.672039] iommu: Default domain type: Passthrough [ 0.674689] SCSI subsystem initialized [ 0.676164] ACPI: bus type USB registered [ 0.678142] usbcore: registered new interface driver usbfs [ 0.681194] usbcore: registered new interface driver hub [ 0.683152] usbcore: registered new device driver usb [ 0.685191] pps_core: LinuxPPS API ver. 1 registered [ 0.687015] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.691073] PTP clock support registered [ 0.693067] EDAC MC: Ver: 3.0.0 [ 0.695303] PCI: Using ACPI for IRQ routing [ 0.696673] NetLabel: Initializing [ 0.698018] NetLabel: domain hash size = 128 [ 0.700056] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.702081] NetLabel: unlabeled traffic allowed by default [ 0.704266] vgaarb: loaded [ 0.706295] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.708011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.717223] clocksource: Switched to clocksource kvm-clock [ 0.826758] VFS: Disk quotas dquot_6.6.0 [ 0.828380] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.830752] *** VALIDATE ramfs *** [ 0.831899] *** VALIDATE hugetlbfs *** [ 0.833419] pnp: PnP ACPI init [ 0.835707] pnp: PnP ACPI: found 6 devices [ 0.851426] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.854728] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.856814] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.858937] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.861362] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.863785] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.866289] NET: Registered protocol family 2 [ 0.868610] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.873682] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.877431] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.882427] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.886705] TCP: Hash tables configured (established 65536 bind 65536) [ 0.889596] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.892700] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.894919] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.897126] NET: Registered protocol family 1 [ 0.899547] RPC: Registered named UNIX socket transport module. [ 0.901772] RPC: Registered udp transport module. [ 0.903781] RPC: Registered tcp transport module. [ 0.905725] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.908680] NET: Registered protocol family 44 [ 0.910249] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.912266] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.913911] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.916157] PCI: CLS 0 bytes, default 64 [ 0.917743] Unpacking initramfs... [ 2.324665] debug: unmapping init [mem 0xffff8c6f3cc54000-0xffff8c6f3ffbffff] [ 2.328784] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.331438] software IO TLB: mapped [mem 0x00000000b8c54000-0x00000000bcc54000] (64MB) [ 2.334714] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.832492] Initialise system trusted keyrings [ 2.835538] Key type blacklist registered [ 2.837370] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.847374] zbud: loaded [ 2.850467] *** VALIDATE nfs *** [ 2.851705] *** VALIDATE nfs4 *** [ 2.853282] pstore: using deflate compression [ 2.858288] Platform Keyring initialized [ 2.966977] NET: Registered protocol family 38 [ 2.968295] Key type asymmetric registered [ 2.969851] Asymmetric key parser 'x509' registered [ 2.971633] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.974902] io scheduler mq-deadline registered [ 2.976549] io scheduler kyber registered [ 2.978111] io scheduler bfq registered [ 2.980122] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.983419] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.986503] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.989701] ACPI: Power Button [PWRF] [ 2.995503] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.002638] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.021712] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.033276] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.058905] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.086523] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.113738] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.119028] Non-volatile memory driver v1.3 [ 3.120612] Linux agpgart interface v0.103 [ 3.153492] virtio_blk virtio1: [vda] 146976 512-byte logical blocks (75.3 MB/71.8 MiB) [ 3.156368] vda: detected capacity change from 0 to 75251712 [ 3.174606] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.179273] vdb: detected capacity change from 0 to 1073741824 [ 3.201589] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.204513] vdc: detected capacity change from 0 to 2621440000 [ 3.223327] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.227314] vdd: detected capacity change from 0 to 2621440000 [ 3.247640] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.250987] vde: detected capacity change from 0 to 4294967296 [ 3.270055] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.272879] vdf: detected capacity change from 0 to 4294967296 [ 3.282105] libphy: Fixed MDIO Bus: probed [ 3.287880] usbcore: registered new interface driver usbserial_generic [ 3.290113] usbserial: USB Serial support registered for generic [ 3.292393] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.296326] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.297811] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.300433] mousedev: PS/2 mouse device common for all mice [ 3.303940] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.305253] rtc_cmos 00:05: RTC can wake from S4 [ 3.314322] rtc_cmos 00:05: registered as rtc0 [ 3.314705] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.316708] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.324104] intel_pstate: CPU model not supported [ 3.325097] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.331174] hid: raw HID events driver (C) Jiri Kosina [ 3.333563] usbcore: registered new interface driver usbhid [ 3.335744] usbhid: USB HID core driver [ 3.337326] drop_monitor: Initializing network drop monitor service [ 3.339405] Initializing XFRM netlink socket [ 3.341254] NET: Registered protocol family 10 [ 3.344756] Segment Routing with IPv6 [ 3.346355] NET: Registered protocol family 17 [ 3.349413] mpls_gso: MPLS GSO support [ 3.356318] RAS: Correctable Errors collector initialized. [ 3.359099] AVX version of gcm_enc/dec engaged. [ 3.361108] AES CTR mode by8 optimization enabled [ 3.443179] sched_clock: Marking stable (3443136785, 0)->(4447675014, -1004538229) [ 3.447374] registered taskstats version 1 [ 3.449638] Loading compiled-in X.509 certificates [ 3.452218] zswap: loaded using pool lzo/zbud [ 3.478936] Key type big_key registered [ 3.491878] Key type encrypted registered [ 3.493710] ima: No TPM chip found, activating TPM-bypass! [ 3.495707] ima: Allocated hash algorithm: sha1 [ 3.497320] ima: No architecture policies found [ 3.499403] evm: Initialising EVM extended attributes: [ 3.501649] evm: security.selinux [ 3.502868] evm: security.ima [ 3.504213] evm: security.capability [ 3.505454] evm: HMAC attrs: 0x1 [ 3.508139] rtc_cmos 00:05: setting system clock to 2026-08-27 21:19:20 UTC (1787865560) [ 3.514573] debug: unmapping init [mem 0xffffffff8b603000-0xffffffff8b7fffff] [ 3.517896] debug: unmapping init [mem 0xffffffff8a382000-0xffffffff8a658fff] [ 3.528137] Write protecting the kernel read-only data: 28672k [ 3.531814] debug: unmapping init [mem 0xffffffff88a03000-0xffffffff88bfffff] [ 3.536038] debug: unmapping init [mem 0xffffffff89314000-0xffffffff893fffff] [ 3.584306] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.594303] systemd[1]: Detected virtualization kvm. [ 3.596447] systemd[1]: Detected architecture x86-64. [ 3.598845] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.624523] systemd[1]: No hostname configured. [ 3.626665] systemd[1]: Set hostname to . [ 3.628905] random: systemd: uninitialized urandom read (16 bytes read) [ 3.631418] systemd[1]: Initializing machine ID from random generator. [ 3.673415] random: ln: uninitialized urandom read (6 bytes read) [ 3.784759] random: systemd: uninitialized urandom read (16 bytes read) [ 3.787751] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.793423] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.803102] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Journal Service... Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Paths. [ OK ] Reached target Timers. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.471752] device-mapper: uevent: version 1.0.3 [ 4.473887] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.177795] random: fast init done [ 5.199349] virtio_net virtio0 ens2: renamed from eth0 [ 5.247420] scsi host0: ata_piix [ 5.290845] scsi host1: ata_piix [ 5.292840] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.295854] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.540496] dracut-initqueue[589]: RTNETLINK answers: File exists [ 10.011448] random: crng init done [ 10.013739] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.427804] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Slices. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.584563] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.860042] SELinux: Disabled at runtime. [ 11.920475] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.928944] systemd[1]: Detected virtualization kvm. [ 11.930805] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.425661] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.429484] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.434141] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.441339] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.444225] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.452037] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.455901] systemd[1]: Reached target rpc_pipefs.target. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice User and Session Slice. Mounting POSIX Message Queue File System... [ OK ] Stopped target Switch Root. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Remount Root and Kernel File Systems... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Create list of required st…ce nodes for the current kernel... Activating swap /dev/disk/by-label/SWAP... Mounting Huge Pages File System... Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. Starting udev Coldplug [ 12.571744] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS all Devices... [ OK ] Reached target Slices. Mounting Kernel Debug File System... [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-getty.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 12.879201] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.183969] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.212812] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.321082] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.337645] EDAC sbridge: Ver: 1.1.2 [ 15.139490] Key type dns_resolver registered [ 15.446607] NFS: Registering the id_resolver key type [ 15.449046] Key type id_resolver registered [ 15.450952] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started irqbalance daemon. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started D-Bus System Message Bus. [ OK ] Reached target sshd-keygen.target. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg426-server login: [ 39.188286] hrtimer: interrupt took 3451809 ns [ 69.038385] libcfs: loading out-of-tree module taints kernel. [ 69.095389] Key type ._llcrypt registered [ 69.097666] Key type .llcrypt registered [ 69.220793] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 87.587061] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 88.848251] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 88.880090] alg: No test for adler32 (adler32-zlib) [ 90.480211] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [ 91.320760] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 93.152175] Key type lgssc registered [ 94.704429] Lustre: Echo OBD driver; http://www.lustre.org/ [ 110.730768] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 144.650934] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 155.401722] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 155.424430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 156.714681] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 156.755575] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 156.852517] Lustre: lustre-MDT0000: new disk, initializing [ 156.911868] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 156.935959] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 161.499902] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 174.849588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 174.951288] Lustre: 6508:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 174.980361] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 174.984872] Lustre: Skipped 1 previous similar message [ 175.049227] Lustre: lustre-MDT0001: new disk, initializing [ 175.110735] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 175.135336] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 175.143168] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 179.329631] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 184.238228] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 193.079400] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 193.402944] Lustre: lustre-OST0000: new disk, initializing [ 193.408822] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 193.414883] Lustre: 8445:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 193.482141] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 196.899425] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 196.909795] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 196.997250] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 199.073844] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 212.501611] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 212.636847] Lustre: lustre-OST0001: new disk, initializing [ 212.641178] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 212.646626] Lustre: 9516:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 212.713213] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 219.174370] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 219.209036] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 219.279814] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 219.512502] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 231.400733] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 237.696439] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 243.727585] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing check_logdir /tmp/testlogs/ [ 248.851652] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing yml_node [ 252.796654] Lustre: DEBUG MARKER: Client: 2.17.57.103 [ 255.165855] Lustre: DEBUG MARKER: MDS: 2.17.57.103 [ 257.879742] Lustre: DEBUG MARKER: OSS: 2.17.57.103 [ 259.821704] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Aug 27 17:23:34 EDT 2026 [ 277.288982] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 279.060939] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 297.958928] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 297.967180] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 297.986604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 301.027454] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 301.030189] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 301.054980] Lustre: Skipped 1 previous similar message [ 301.078761] Lustre: Skipped 3 previous similar messages [ 302.356556] Lustre: server umount lustre-MDT0000 complete [ 306.605536] LustreError: 6499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787865863 with bad export cookie 6053645001875620770 [ 306.615667] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 306.617476] LustreError: 6499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 311.266467] LustreError: 6515:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 311.268484] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 311.296211] LustreError: 6515:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 6 previous similar messages [ 311.331910] Lustre: Skipped 1 previous similar message [ 311.342231] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 311.354266] Lustre: Skipped 1 previous similar message [ 312.969635] Lustre: server umount lustre-MDT0001 complete [ 317.281824] Lustre: server umount lustre-OST0000 complete [ 321.466430] Lustre: server umount lustre-OST0001 complete [ 328.459726] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 337.973750] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 371.612992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 371.864292] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 371.909223] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 371.963490] Lustre: lustre-MDT0000: new disk, initializing [ 372.041434] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 372.056144] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 376.662577] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 386.521703] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 386.651277] Lustre: 15070:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 386.700193] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 386.705748] Lustre: Skipped 1 previous similar message [ 386.796770] Lustre: lustre-MDT0001: new disk, initializing [ 386.898729] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 386.933732] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 386.944134] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 391.253494] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 400.518644] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 406.453681] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 413.237769] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 413.627881] Lustre: lustre-OST0000: new disk, initializing [ 413.639136] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 413.652997] Lustre: 17037:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 413.772438] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 415.433255] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 415.440851] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 415.572957] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 419.889265] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 429.714345] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 433.799717] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 433.962187] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 438.464304] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 438.691859] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 441.323631] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 441.325045] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 441.330496] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 441.337051] LustreError: Skipped 1 previous similar message [ 441.351992] Lustre: Skipped 2 previous similar messages [ 446.437634] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 446.449784] Lustre: Skipped 2 previous similar messages [ 447.335954] Lustre: server umount lustre-OST0000 complete [ 453.603212] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 453.611904] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 456.679910] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 456.686622] Lustre: Skipped 2 previous similar messages [ 466.400140] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 466.637748] Lustre: server umount lustre-MDT0000 complete [ 466.915781] LustreError: 15082:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 466.940320] LustreError: 15082:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 470.641071] LustreError: 17048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787866027 with bad export cookie 6053645001875622296 [ 470.642816] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 470.658752] LustreError: 17048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 471.023455] Lustre: server umount lustre-MDT0001 complete [ 476.124749] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 17:27:10 (1787866030) [ 479.528567] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 481.199058] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 17:27:16 (1787866036) [ 488.617023] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 489.342399] LustreError: 19450:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 489.445719] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 494.429509] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 494.571318] LustreError: 19451:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 499.683354] LustreError: 19450:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 504.459059] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 508.836724] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 516.354920] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 522.270844] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 530.726916] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 538.150120] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 547.411710] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 560.111726] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 560.118331] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 560.137188] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 560.139469] Lustre: Skipped 2 previous similar messages [ 560.152385] Lustre: Skipped 2 previous similar messages [ 564.638538] Lustre: server umount lustre-OST0000 complete [ 574.907543] Lustre: server umount lustre-MDT0000 complete [ 575.464659] LustreError: 19450:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 575.500684] LustreError: 19450:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 578.955468] LustreError: 19430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787866135 with bad export cookie 6053645001875623185 [ 578.963084] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 578.972879] LustreError: 19430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 579.341557] Lustre: server umount lustre-MDT0001 complete [ 591.979750] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 594.731657] Key type lgssc unregistered [ 594.981437] LNet: 22987:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 594.988207] LNetError: 22987:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 596.008248] LNet: Removed LNI 192.168.204.126@tcp [ 596.930156] Key type .llcrypt unregistered [ 596.932174] Key type ._llcrypt unregistered [ 623.041694] Key type ._llcrypt registered [ 623.044497] Key type .llcrypt registered [ 623.165829] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 624.155129] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 624.175345] alg: No test for adler32 (adler32-zlib) [ 625.231537] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [ 625.481248] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 627.137636] Key type lgssc registered [ 628.404505] Lustre: Echo OBD driver; http://www.lustre.org/ [ 636.553199] loop: module loaded [ 636.712772] loop0: detected capacity change from 0 to 204800000 [ 637.019298] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 637.035469] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 637.216863] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 644.194437] loop0: detected capacity change from 0 to 204800000 [ 644.377930] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 644.388952] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 644.414728] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 650.092872] loop0: detected capacity change from 0 to 204800000 [ 656.465816] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 656.490377] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 657.814085] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 657.902600] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 658.007845] Lustre: lustre-MDT0000: new disk, initializing [ 658.162680] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 658.198830] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 662.582578] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 672.304611] loop1: detected capacity change from 0 to 204800000 [ 678.774500] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 679.159838] Lustre: lustre-OST07c6: new disk, initializing [ 679.165202] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 679.170851] Lustre: Skipped 1 previous similar message [ 679.177514] Lustre: 26178:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 679.266227] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 685.300360] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 689.231397] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 689.239847] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 689.284134] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 695.471884] Lustre: 26695:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 695.491488] Lustre: 26695:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 695.502265] Lustre: 26695:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 695.510294] Lustre: 26695:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 695.515981] Lustre: 26695:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 695.522351] Lustre: 26695:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 717.015051] Lustre: 25161:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 717.029519] Lustre: 25161:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 717.043048] Lustre: 25161:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 717.055540] Lustre: 25161:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 717.078406] Lustre: 25161:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 717.087423] Lustre: 25161:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 722.484540] Lustre: 25163:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 722.496702] Lustre: 25163:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 722.507709] Lustre: 25163:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 722.518175] Lustre: 25163:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 722.531692] Lustre: 25163:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 722.537676] Lustre: 25163:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 722.542351] Lustre: 25163:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 722.547218] Lustre: 25163:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 722.551271] Lustre: 25163:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 722.555584] Lustre: 25163:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 722.562523] Lustre: 25163:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 722.567460] Lustre: 25163:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 727.504392] Lustre: 26695:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 727.513068] Lustre: 26695:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 727.522116] Lustre: 26695:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 727.539433] Lustre: 26695:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 727.551116] Lustre: 26695:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 727.559659] Lustre: 26695:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 727.567022] Lustre: 26695:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 727.574018] Lustre: 26695:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 727.578196] Lustre: 26695:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 727.587138] Lustre: 26695:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 727.598839] Lustre: 26695:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 727.611477] Lustre: 26695:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 732.656949] Lustre: 25162:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 732.667551] Lustre: 25162:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 732.671950] Lustre: 25162:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 732.681180] Lustre: 25162:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 732.693282] Lustre: 25162:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 732.705209] Lustre: 25162:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 732.717483] Lustre: 25162:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 732.723024] Lustre: 25162:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 732.727515] Lustre: 25162:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 732.737561] Lustre: 25162:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 732.750179] Lustre: 25162:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 732.756641] Lustre: 25162:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 740.876873] Lustre: 25162:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 740.886250] Lustre: 25162:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages [ 740.892833] Lustre: 25162:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 740.898821] Lustre: 25162:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 740.913441] Lustre: 25162:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 740.923356] Lustre: 25162:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 740.930354] Lustre: 25162:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 740.938740] Lustre: 25162:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 740.945419] Lustre: 25162:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 740.958500] Lustre: 25162:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 740.970349] Lustre: 25162:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 740.980252] Lustre: 25162:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 796.642894] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 796.657787] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 796.668909] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 799.718315] Lustre: server umount lustre-OST07c6 complete [ 810.255870] Lustre: server umount lustre-MDT0000 complete [ 829.332982] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 832.071235] Key type lgssc unregistered [ 832.427700] LNet: 28743:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 832.442869] LNetError: 28743:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 832.468522] LNet: Removed LNI 192.168.204.126@tcp [ 833.371384] Key type .llcrypt unregistered [ 833.372781] Key type ._llcrypt unregistered [ 854.341683] Key type ._llcrypt registered [ 854.344876] Key type .llcrypt registered [ 854.450687] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 856.033575] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 856.135395] alg: No test for adler32 (adler32-zlib) [ 857.271672] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [ 857.575937] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 859.336161] Key type lgssc registered [ 860.532582] Lustre: Echo OBD driver; http://www.lustre.org/ [ 869.436491] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 869.488149] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 871.040927] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 875.968325] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 881.637246] LustreError: 30013:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 881.651888] LustreError: 30013:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 884.003951] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 884.422804] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 889.209976] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 898.211270] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 905.739350] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 912.916871] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 913.206452] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 916.286717] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 920.163502] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 929.150561] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 947.169455] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 947.177436] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 947.188557] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 947.592469] Lustre: server umount lustre-OST0000 complete [ 955.365914] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 955.369958] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 955.378160] Lustre: Skipped 1 previous similar message [ 955.392644] Lustre: Skipped 1 previous similar message [ 960.481341] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 960.486705] Lustre: Skipped 1 previous similar message [ 965.610321] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 965.621059] Lustre: Skipped 1 previous similar message [ 966.113816] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 966.314701] Lustre: server umount lustre-MDT0000 complete [ 966.624181] Lustre: 29324:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787866507/real 1787866507] req@000000008cb4d10e x1874713812959232/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787866523 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 966.644914] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 966.655201] LustreError: 30741:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 966.672993] LustreError: 30741:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 970.078248] LustreError: 33000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787866527 with bad export cookie 15632862435440627216 [ 970.091236] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 970.092157] LustreError: 33000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 970.470772] Lustre: server umount lustre-MDT0001 complete [ 981.306413] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 983.760772] Key type lgssc unregistered [ 984.020626] LNet: 33756:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 984.026557] LNetError: 33756:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 984.040083] LNet: Removed LNI 192.168.204.126@tcp [ 984.881345] Key type .llcrypt unregistered [ 984.887069] Key type ._llcrypt unregistered [ 1002.279654] Key type ._llcrypt registered [ 1002.281268] Key type .llcrypt registered [ 1002.391176] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 1003.219882] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1003.284861] alg: No test for adler32 (adler32-zlib) [ 1004.337910] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [ 1004.525417] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 1006.160172] Key type lgssc registered [ 1007.106735] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1014.269178] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1014.286716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1015.795444] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1020.331432] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1026.021666] LustreError: 35028:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1026.039083] LustreError: 35028:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1028.282789] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1028.647689] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1033.240983] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1043.130380] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1050.455769] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1059.694377] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1060.322712] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1061.300042] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1067.708897] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1076.211252] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1088.994307] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1089.000721] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1089.007185] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1089.039366] Lustre: Skipped 1 previous similar message [ 1093.900889] Lustre: server umount lustre-OST0000 complete [ 1099.236502] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1099.269744] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1099.282666] Lustre: Skipped 2 previous similar messages [ 1104.357163] LustreError: 35022:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1104.385136] LustreError: 35022:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1105.979977] Lustre: server umount lustre-MDT0000 complete [ 1109.476634] LustreError: 35027:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1109.500472] LustreError: 35027:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1109.862439] LustreError: 35010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787866666 with bad export cookie 12339239851648930189 [ 1109.871790] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1109.873632] LustreError: 35010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1110.291216] Lustre: server umount lustre-MDT0001 complete [ 1120.953857] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 1123.926863] Key type lgssc unregistered [ 1124.260485] LNet: 38566:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1124.274164] LNetError: 38566:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1124.301970] LNet: Removed LNI 192.168.204.126@tcp [ 1125.245767] Key type .llcrypt unregistered [ 1125.252098] Key type ._llcrypt unregistered [ 1147.963843] Key type ._llcrypt registered [ 1147.966700] Key type .llcrypt registered [ 1148.099419] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 1149.369612] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1149.424519] alg: No test for adler32 (adler32-zlib) [ 1150.458836] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [ 1150.763647] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 1152.472387] Key type lgssc registered [ 1153.558898] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1161.359958] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1161.378245] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1162.997457] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1167.734042] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1172.459860] LustreError: 39982:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1172.477355] LustreError: 39982:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1177.120647] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1177.721461] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1182.445578] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1190.621201] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1196.540334] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1203.831845] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1204.080800] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1207.167637] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1210.961467] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1219.378814] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1232.844832] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.26@tcp (stopping) [ 1232.867463] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1232.888973] Lustre: Skipped 2 previous similar messages [ 1236.305443] Lustre: server umount lustre-MDT0000 complete [ 1237.940536] LustreError: 41820:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1239.822961] LustreError: 39963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787866796 with bad export cookie 10645551956888873691 [ 1239.828105] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1239.840498] LustreError: 39963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1243.061614] LustreError: 39977:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1243.090288] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.26@tcp (stopping) [ 1243.101898] LustreError: 39977:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 1243.104133] Lustre: Skipped 3 previous similar messages [ 1243.111058] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1244.130974] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1244.139175] Lustre: Skipped 1 previous similar message [ 1248.176502] LustreError: 39978:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1248.177082] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.26@tcp (stopping) [ 1248.205346] LustreError: 39978:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1253.300244] LustreError: 39979:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1253.300268] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.26@tcp (stopping) [ 1253.300274] Lustre: Skipped 1 previous similar message [ 1253.320104] LustreError: 39979:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1254.368295] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1254.659322] Lustre: server umount lustre-MDT0001 complete [ 1269.536042] Lustre: server umount lustre-OST0000 complete [ 1285.313360] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 1288.544788] Key type lgssc unregistered [ 1288.861601] LNet: 43848:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1288.873879] LNetError: 43848:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1288.894945] LNet: Removed LNI 192.168.204.126@tcp [ 1289.886461] Key type .llcrypt unregistered [ 1289.888493] Key type ._llcrypt unregistered [ 1312.957592] Key type ._llcrypt registered [ 1312.959968] Key type .llcrypt registered [ 1313.049229] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 1314.358801] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1314.372099] alg: No test for adler32 (adler32-zlib) [ 1315.388784] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [ 1315.614143] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 1317.297238] Key type lgssc registered [ 1318.800759] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1326.402201] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1326.413364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1327.786120] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1332.406469] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1333.223458] LustreError: 45264:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1333.237553] LustreError: 45264:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1337.316689] LustreError: 45265:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1340.991538] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1341.423354] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1345.392210] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1353.342402] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1359.867542] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1367.081519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1367.406407] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1372.561622] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1375.576820] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1386.812299] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1393.123926] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1393.135722] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1393.146405] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1393.160159] Lustre: Skipped 1 previous similar message [ 1398.048610] Lustre: server umount lustre-OST0000 complete [ 1418.209512] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1418.218839] LustreError: Skipped 1 previous similar message [ 1418.227747] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1418.240367] Lustre: Skipped 1 previous similar message [ 1418.245814] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1418.250991] Lustre: Skipped 1 previous similar message [ 1420.843654] Lustre: server umount lustre-MDT0000 complete [ 1423.843720] LustreError: 45265:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1423.858461] LustreError: 45265:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1424.122289] LustreError: 45244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787866981 with bad export cookie 8559182191182044932 [ 1424.123630] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1424.128798] LustreError: 45244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1430.423820] Lustre: server umount lustre-MDT0001 complete [ 1442.551372] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 1445.784359] Key type lgssc unregistered [ 1446.126709] LNet: 48978:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1446.136739] LNetError: 48978:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1446.159276] LNet: Removed LNI 192.168.204.126@tcp [ 1446.901989] Key type .llcrypt unregistered [ 1446.905906] Key type ._llcrypt unregistered [ 1468.194654] Key type ._llcrypt registered [ 1468.196487] Key type .llcrypt registered [ 1468.287822] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 1469.600931] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1469.748734] alg: No test for adler32 (adler32-zlib) [ 1470.857244] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [ 1471.155881] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 1472.856238] Key type lgssc registered [ 1474.326752] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1482.897705] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1482.920798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1484.320538] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1489.112559] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1494.512775] LustreError: 50398:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1494.541811] LustreError: 50398:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1498.921332] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1499.296520] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1503.348152] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1511.839445] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1517.932402] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1525.511379] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1525.915465] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1530.033964] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1533.361710] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1540.841665] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1550.506787] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1554.006383] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1566.681190] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-2717h45m20s [ 1568.522404] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1570.787441] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1570.799724] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1570.816900] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1571.875714] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1571.878897] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1571.886336] Lustre: Skipped 1 previous similar message [ 1571.896810] Lustre: Skipped 2 previous similar messages [ 1572.911272] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1572.925539] Lustre: Skipped 1 previous similar message [ 1574.951253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1574.956909] Lustre: Skipped 3 previous similar messages [ 1576.771543] Lustre: server umount lustre-MDT0000 complete [ 1577.012155] LustreError: 50393:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1577.028778] LustreError: 50393:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1578.024745] LustreError: 50398:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1580.085619] LustreError: 52406:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1580.098893] LustreError: 52406:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1580.415652] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1584.169249] LustreError: 52406:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1584.181706] LustreError: 52406:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 6 previous similar messages [ 1587.259133] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1587.406475] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1587.647166] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1587.746684] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1589.692417] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1592.222284] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1595.643851] Lustre: DEBUG MARKER: Wait for df (36679) ... [ 1597.047704] Lustre: DEBUG MARKER: done [ 1600.994307] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1600.999720] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1601.010880] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1601.013294] Lustre: Skipped 2 previous similar messages [ 1603.041072] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1603.049517] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1606.630664] Lustre: server umount lustre-OST0000 complete [ 1610.216563] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1610.228083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1610.233879] Lustre: Skipped 1 previous similar message [ 1610.241138] Lustre: Skipped 3 previous similar messages [ 1616.119428] Lustre: server umount lustre-MDT0000 complete [ 1616.354051] LustreError: 50397:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1616.368774] LustreError: 50397:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 6 previous similar messages [ 1619.546890] LustreError: 50378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787867176 with bad export cookie 10181945097333445071 [ 1619.551710] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1619.557683] LustreError: 50378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1619.808952] Lustre: server umount lustre-MDT0001 complete [ 1628.891760] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 1631.420728] Key type lgssc unregistered [ 1631.643680] LNet: 55324:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1631.651157] LNetError: 55324:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1631.661908] LNet: Removed LNI 192.168.204.126@tcp [ 1632.341108] Key type .llcrypt unregistered [ 1632.343370] Key type ._llcrypt unregistered [ 1666.649857] Key type ._llcrypt registered [ 1666.651330] Key type .llcrypt registered [ 1666.730246] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 1667.458463] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1667.478169] alg: No test for adler32 (adler32-zlib) [ 1668.575835] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [ 1668.853722] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 1670.552222] Key type lgssc registered [ 1671.702287] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1680.332788] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1680.373821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1681.748716] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1681.780378] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1682.083248] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1688.056740] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1698.888654] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1699.090646] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1699.165855] Lustre: 57534:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1699.411308] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1704.274664] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1712.604924] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1718.827635] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1726.579625] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1726.718097] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1726.893942] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1733.207835] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1736.899364] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1740.721491] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1746.813648] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-2717h48m21s [ 1748.337577] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1755.645049] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1757.772180] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1757.832519] LustreError: 57542:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1778.137572] Lustre: lustre-MDT0000: Client 07eb759d-e304-4695-8a91-60e0657de7b3 (at 192.168.204.26@tcp) reconnecting [ 1797.904146] LustreError: 57542:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1800.267337] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1803.070232] Lustre: DEBUG MARKER: done [ 1811.378947] Lustre: server umount lustre-OST0000 complete [ 1817.059512] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1817.074571] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1817.084947] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1818.153531] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1818.161309] Lustre: Skipped 2 previous similar messages [ 1824.228074] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1825.323494] Lustre: server umount lustre-MDT0000 complete [ 1829.344169] Lustre: 56857:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787867370/real 1787867370] req@0000000089fe5442 x1874714663377280/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787867386 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1829.372565] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1829.388403] Lustre: Skipped 1 previous similar message [ 1829.633510] LustreError: 57527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787867386 with bad export cookie 9962197143792891650 [ 1829.642619] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1829.914175] Lustre: server umount lustre-MDT0001 complete [ 1840.209723] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 1842.517526] Key type lgssc unregistered [ 1842.754577] LNet: 61869:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1842.762668] LNetError: 61869:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1842.775035] LNet: Removed LNI 192.168.204.126@tcp [ 1843.579951] Key type .llcrypt unregistered [ 1843.585515] Key type ._llcrypt unregistered [ 1893.024050] Key type ._llcrypt registered [ 1893.027226] Key type .llcrypt registered [ 1893.162794] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 1894.008235] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1894.084266] alg: No test for adler32 (adler32-zlib) [ 1895.277745] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [ 1895.485534] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 1897.145565] Key type lgssc registered [ 1898.221875] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1901.206837] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1901.245826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1902.554883] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1902.594476] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1902.780340] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1903.821688] Lustre: Failing over lustre-MDT0000 [ 1904.166888] Lustre: server umount lustre-MDT0000 complete [ 1911.788448] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 17:51:06 (1787867466) [ 1919.368277] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1919.747541] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1924.552912] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1933.463733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1933.596256] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1933.647914] Lustre: 64671:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1933.892780] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1938.360080] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1946.687761] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1953.100383] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1960.243904] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1960.528771] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1960.747857] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1967.696124] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1970.328132] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1976.306835] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1982.768807] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1989.280729] Lustre: server umount lustre-MDT0000 complete [ 1990.123255] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1990.132815] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1991.145286] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1993.544553] LustreError: 64663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787867550 with bad export cookie 14625127280631335328 [ 1993.551905] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1993.556894] LustreError: 64663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1993.934618] Lustre: server umount lustre-MDT0001 complete [ 1997.377376] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2001.705127] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2010.091656] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2012.448373] Lustre: 63592:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787867553/real 1787867553] req@000000004a2c4eb7 x1874714901388032/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787867569 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2012.479301] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2012.492463] Lustre: Skipped 1 previous similar message [ 2014.689699] Lustre: 63594:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787867555/real 1787867555] req@000000004fe04524 x1874714901388160/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787867571 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2017.824198] Lustre: 63594:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787867558/real 1787867558] req@0000000068c49b02 x1874714901388288/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787867574 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2018.656792] LustreError: 63591:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000002affc18e x1874714901389184/t0(0) o250->MGC192.168.204.126@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2018.944715] LustreError: 66516:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2018.965978] LustreError: 66516:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [ 2019.058668] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2019.149730] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2020.000228] Lustre: 63593:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787867560/real 1787867560] req@000000008e3c03ec x1874714901388544/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787867576 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2022.115578] LustreError: 68439:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2022.132114] LustreError: 68439:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2024.031891] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2024.162633] Lustre: 63593:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787867565/real 1787867565] req@00000000f535a500 x1874714901388672/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787867581 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2027.491514] LustreError: 66517:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2027.515890] LustreError: 66517:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2031.602367] LustreError: 68439:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2031.607167] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2031.616259] LustreError: 68439:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2033.005589] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2033.509193] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2038.681199] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2038.768967] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2047.562117] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2054.638320] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2067.476379] Lustre: server umount lustre-MDT0000 complete [ 2069.476682] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2069.483450] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2069.516712] Lustre: Skipped 2 previous similar messages [ 2069.540200] LustreError: 68437:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2069.575161] LustreError: 68437:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2074.048492] LustreError: 67580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787867631 with bad export cookie 14625127280631337603 [ 2074.060561] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2074.078251] LustreError: 67580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2074.594820] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2074.710226] Lustre: server umount lustre-MDT0001 complete [ 2082.860121] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2093.394450] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2099.443028] LustreError: 66516:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2099.462434] LustreError: 66516:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2099.547098] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2099.619683] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2103.647843] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2114.757268] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2120.699102] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2121.290154] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2124.846496] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2129.614775] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2135.457831] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2142.294510] Lustre: server umount lustre-MDT0000 complete [ 2145.259151] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2145.260414] LustreError: 69173:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2145.280706] Lustre: Skipped 1 previous similar message [ 2145.300026] LustreError: 69173:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 2145.860373] LustreError: 64662:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787867702 with bad export cookie 14625127280631338982 [ 2145.864166] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2145.872118] LustreError: 64662:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2146.292160] Lustre: server umount lustre-MDT0001 complete [ 2153.259855] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2157.378719] Lustre: DEBUG MARKER: files compared the same [ 2169.941402] Lustre: server umount lustre-OST0000 complete [ 2187.922267] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 2191.151558] Key type lgssc unregistered [ 2191.564689] LNet: 74396:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2191.584565] LNetError: 74396:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2191.601241] LNet: Removed LNI 192.168.204.126@tcp [ 2192.446521] Key type .llcrypt unregistered [ 2192.448948] Key type ._llcrypt unregistered [ 2219.129659] Key type ._llcrypt registered [ 2219.134265] Key type .llcrypt registered [ 2219.320452] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 2220.886871] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2221.059356] alg: No test for adler32 (adler32-zlib) [ 2222.103627] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [ 2222.411877] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 2224.153934] Key type lgssc registered [ 2225.900929] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2236.790661] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2236.834065] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2238.333473] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2243.687838] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug +malloc all [ 2248.680455] LustreError: 75823:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2248.693083] LustreError: 75823:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2252.443087] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2252.878608] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2258.376512] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug +malloc all [ 2266.436116] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2272.045706] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2279.260226] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2279.559095] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2283.653082] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2287.302077] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug +malloc all [ 2297.461794] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2309.601100] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2309.617730] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2309.639779] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2312.456271] Lustre: server umount lustre-OST0000 complete [ 2319.329496] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2319.333603] LustreError: Skipped 1 previous similar message [ 2319.336629] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2319.347185] Lustre: Skipped 1 previous similar message [ 2319.351754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2319.356987] Lustre: Skipped 1 previous similar message [ 2324.964255] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2324.969631] Lustre: Skipped 2 previous similar messages [ 2329.062839] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2329.074107] Lustre: Skipped 1 previous similar message [ 2331.104131] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2331.252357] Lustre: server umount lustre-MDT0000 complete [ 2334.180097] LustreError: 75824:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2334.895501] LustreError: 75804:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787867891 with bad export cookie 13398116154976108114 [ 2334.909120] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2335.073434] Lustre: server umount lustre-MDT0001 complete [ 2347.483889] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 2351.034375] Key type lgssc unregistered [ 2351.286652] LNet: 79362:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2351.293047] LNetError: 79362:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2351.308859] LNet: Removed LNI 192.168.204.126@tcp [ 2352.237483] Key type .llcrypt unregistered [ 2352.239929] Key type ._llcrypt unregistered [ 2379.123473] Key type ._llcrypt registered [ 2379.127241] Key type .llcrypt registered [ 2379.202148] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 2380.189643] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2380.237779] alg: No test for adler32 (adler32-zlib) [ 2381.373787] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [ 2381.701971] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 2383.376548] Key type lgssc registered [ 2384.558808] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2393.717343] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2393.781129] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2409.440465] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2414.560325] LustreError: 80754:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.126@tcp: failed processing log, type 4: rc = -110 [ 2441.312732] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2448.829502] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2457.716742] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2466.559117] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2466.640698] Lustre: *** cfs_fail_loc=706, val=0*** [ 2467.040245] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2467.110295] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2472.429177] LustreError: 80780:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2472.449936] LustreError: 80780:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 2473.135189] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2476.581145] LustreError: 80779:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2476.587813] LustreError: 80779:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2481.645225] LustreError: 80780:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2481.655188] LustreError: 80780:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2483.981975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2484.546445] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2490.562313] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2500.414800] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2507.380547] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2510.332266] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2510.335826] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2510.365924] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2515.432251] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2515.439561] Lustre: Skipped 2 previous similar messages [ 2515.935598] Lustre: server umount lustre-OST0000 complete [ 2520.547369] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2520.555932] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2520.570439] Lustre: Skipped 2 previous similar messages [ 2520.591340] Lustre: Skipped 1 previous similar message [ 2525.670181] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2525.679956] Lustre: Skipped 1 previous similar message [ 2526.482230] Lustre: server umount lustre-MDT0000 complete [ 2530.207319] LustreError: 83632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787868087 with bad export cookie 13469370235338010955 [ 2530.214615] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2530.220091] LustreError: 83632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2530.623135] Lustre: server umount lustre-MDT0001 complete [ 2541.870920] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 2545.251930] Key type lgssc unregistered [ 2545.639871] LNet: 84388:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2545.651346] LNetError: 84388:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2545.691894] LNet: Removed LNI 192.168.204.126@tcp [ 2546.602830] Key type .llcrypt unregistered [ 2546.605153] Key type ._llcrypt unregistered [ 2576.444763] Key type ._llcrypt registered [ 2576.451784] Key type .llcrypt registered [ 2576.645460] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 2578.275230] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2578.339812] alg: No test for adler32 (adler32-zlib) [ 2579.507760] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [ 2579.847284] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 2581.664209] Key type lgssc registered [ 2582.915746] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2591.960763] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2592.012924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2597.835390] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2604.697303] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2605.179395] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2609.536055] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2610.670676] LustreError: 86500:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2610.705149] LustreError: 86500:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 2615.778111] LustreError: 86499:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2615.809275] LustreError: 86499:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2618.164820] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2618.486764] LustreError: 86504:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2618.504077] LustreError: 86504:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2618.521364] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2623.970262] LustreError: 86503:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2624.000337] LustreError: 86503:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2624.963406] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2629.093547] LustreError: 86504:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2629.106395] LustreError: 86504:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2633.505249] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2633.512699] Lustre: MGS: Client 4a1d369b-daac-498e-8e28-b2a1a83a8e24 (at 0@lo) reconnecting [ 2633.535277] LustreError: 86514:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2633.728080] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2633.814474] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2639.774564] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2655.715979] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2655.722158] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2655.748825] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2659.307278] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2659.310262] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2659.311517] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2660.839073] Lustre: server umount lustre-OST0000 complete [ 2669.538409] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2669.540876] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2669.552688] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2669.574947] Lustre: Skipped 2 previous similar messages [ 2674.660719] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2679.780537] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2679.801151] Lustre: Skipped 2 previous similar messages [ 2680.289505] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2680.409598] Lustre: server umount lustre-MDT0000 complete [ 2684.085226] Lustre: server umount lustre-MDT0001 complete [ 2687.305391] Lustre: server umount MGS complete [ 2699.827633] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 18:04:14 (1787868254) [ 2733.657589] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 2743.819588] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 2786.722938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2792.335800] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2804.312891] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2804.700860] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2804.763228] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2804.865076] Lustre: lustre-MDT0001: new disk, initializing [ 2804.940983] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2804.989707] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2805.004984] Lustre: 93694:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2805.016191] LustreError: 93694:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 1, retries 0, failed: rc = -115 [ 2810.579615] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2822.260795] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2822.604284] Lustre: lustre-OST0000: new disk, initializing [ 2822.608324] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2822.624798] Lustre: 94460:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2822.743853] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2830.926804] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2834.017122] Lustre: 94919:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000090379f84 x1874715618616960/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787868407 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2841.928449] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2849.030833] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2849.040121] Lustre: MGS: Client 176e7563-f7c4-4ff4-927e-92faf8179460 (at 0@lo) reconnecting [ 2849.125567] Lustre: 95508:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2849.145479] LustreError: 93680:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2849.152709] LustreError: 93680:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2849.184739] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2849.277974] Lustre: lustre-MDT0000: new disk, initializing [ 2849.448146] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2849.466351] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2849.506748] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2850.546415] Lustre: 95532:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000b49fc861 x1874715618623872/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1787868423 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2854.432388] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2855.782725] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2859.705403] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2859.728995] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2859.855520] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2885.097906] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2885.100099] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2885.126265] Lustre: Skipped 2 previous similar messages [ 2885.141931] Lustre: Skipped 1 previous similar message [ 2890.950290] Lustre: server umount lustre-OST0000 complete [ 2895.331577] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2895.337142] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2900.846101] Lustre: server umount lustre-MDT0000 complete [ 2905.575774] LustreError: 96070:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2905.582851] LustreError: 96070:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 2912.912745] Lustre: server umount lustre-MDT0001 complete [ 2918.396877] Lustre: server umount MGS complete [ 2933.077797] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 18:08:07 (1787868487) [ 2954.961520] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 2958.157194] Key type lgssc unregistered [ 2958.482840] LNet: 98037:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2958.489741] LNetError: 98037:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2959.529081] LNet: Removed LNI 192.168.204.126@tcp [ 2960.611990] Key type .llcrypt unregistered [ 2960.614429] Key type ._llcrypt unregistered [ 2986.549909] Key type ._llcrypt registered [ 2986.553623] Key type .llcrypt registered [ 2986.735756] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 2987.887569] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2987.927680] alg: No test for adler32 (adler32-zlib) [ 2989.110319] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [ 2989.428725] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 2991.184195] Key type lgssc registered [ 2992.446462] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3001.289896] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3001.333271] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3002.891288] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3008.345350] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3012.071925] LustreError: 99262:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3012.085909] LustreError: 99262:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3017.197417] LustreError: 99261:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3018.235775] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3018.645674] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3023.398253] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3031.062826] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3037.446892] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3045.935621] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3046.526343] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3047.559061] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3054.298413] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3062.923137] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3073.525140] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3073.534289] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3073.544737] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3074.535498] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3074.540689] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3074.561429] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3074.613977] Lustre: Skipped 1 previous similar message [ 3078.471714] Lustre: server umount lustre-OST0000 complete [ 3084.770549] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3084.784278] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3084.790113] Lustre: Skipped 1 previous similar message [ 3084.805568] Lustre: Skipped 1 previous similar message [ 3089.273112] Lustre: server umount lustre-MDT0000 complete [ 3089.889719] LustreError: 100264:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3094.136663] LustreError: 99241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787868651 with bad export cookie 13068137308241827823 [ 3094.137860] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3094.149227] LustreError: 99241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3094.627424] Lustre: server umount lustre-MDT0001 complete [ 3107.321815] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 3110.574180] Key type lgssc unregistered [ 3110.867874] LNet: 102798:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3110.881233] LNetError: 102798:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3111.920621] LNet: Removed LNI 192.168.204.126@tcp [ 3113.081198] Key type .llcrypt unregistered [ 3113.082828] Key type ._llcrypt unregistered [ 3131.335725] Key type ._llcrypt registered [ 3131.338281] Key type .llcrypt registered [ 3131.437269] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 3132.885792] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3132.928560] alg: No test for adler32 (adler32-zlib) [ 3134.151286] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [ 3134.522688] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 3136.320490] Key type lgssc registered [ 3137.838438] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3145.163333] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3145.168519] LustreError: 103880:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3145.179431] LustreError: 103884:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3145.185833] LustreError: 103880:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4987 [ 3145.195503] LustreError: 103880:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3145.216814] LustreError: 103880:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3145.256140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3146.831155] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3156.289723] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3156.488303] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3161.862216] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3168.606969] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing lsmod [ 3171.948604] LustreError: 105092:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3171.949642] LustreError: 105091:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3171.957919] LustreError: 105092:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4998 [ 3171.962341] LustreError: 105092:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3171.966825] LustreError: 105092:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3171.982672] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3172.358408] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3173.368660] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3176.929619] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3176.943814] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3176.958323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3178.468263] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3178.473621] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3178.487254] Lustre: Skipped 1 previous similar message [ 3178.509400] Lustre: Skipped 1 previous similar message [ 3181.605751] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3181.614639] Lustre: Skipped 3 previous similar messages [ 3182.318385] Lustre: server umount lustre-MDT0000 complete [ 3186.133661] LustreError: 103902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787868743 with bad export cookie 12026920379930508113 [ 3186.141987] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3186.149026] LustreError: 103902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3186.659498] LustreError: 103919:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3186.659743] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3186.670527] LustreError: 103919:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3186.686923] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3187.810210] LustreError: 104353:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000a70bcdc0 x1874716200556800/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3188.019590] Lustre: server umount lustre-MDT0001 complete [ 3193.140902] Lustre: server umount lustre-OST0000 complete [ 3203.430723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3204.214262] LustreError: 106156:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3204.384739] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3209.702767] LustreError: 106157:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3210.306565] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3214.831943] LustreError: 106156:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3219.693282] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3220.074893] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3224.641773] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3233.282060] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3239.303802] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3247.013814] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3247.336457] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3249.427520] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97) [ 3254.918419] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3263.529537] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3276.261338] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3276.262360] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3276.268729] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3276.289642] Lustre: Skipped 1 previous similar message [ 3282.140275] Lustre: server umount lustre-OST0000 complete [ 3291.620565] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3291.621350] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3291.637463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3291.650452] Lustre: Skipped 4 previous similar messages [ 3301.344147] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3301.650074] Lustre: server umount lustre-MDT0000 complete [ 3301.866493] LustreError: 106152:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3301.891092] LustreError: 106152:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3306.020357] LustreError: 106137:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787868863 with bad export cookie 12026920379930509044 [ 3306.024764] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3306.033780] LustreError: 106137:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3306.488801] Lustre: server umount lustre-MDT0001 complete [ 3319.006437] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 3322.171525] Key type lgssc unregistered [ 3322.505167] LNet: 109714:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3322.540124] LNetError: 109714:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3323.569545] LNet: Removed LNI 192.168.204.126@tcp [ 3324.656316] Key type .llcrypt unregistered [ 3324.659446] Key type ._llcrypt unregistered [ 3350.405462] Key type ._llcrypt registered [ 3350.416600] Key type .llcrypt registered [ 3350.531790] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 3351.500422] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3351.517392] alg: No test for adler32 (adler32-zlib) [ 3352.801683] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [ 3353.085964] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 3354.752323] Key type lgssc registered [ 3356.137527] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3364.971493] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3364.984854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3366.505490] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3371.357093] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3376.617900] LustreError: 111134:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3376.648550] LustreError: 111134:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3381.733065] LustreError: 111135:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3382.185484] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3382.854535] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3388.864829] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3398.624895] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3405.995825] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3415.056274] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3415.470348] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3422.556628] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3432.358074] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3436.997697] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3442.070489] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3446.457500] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3452.385578] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3452.398335] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3452.404091] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3452.418094] Lustre: Skipped 1 previous similar message [ 3452.430753] Lustre: Skipped 1 previous similar message [ 3457.505645] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3457.511829] Lustre: Skipped 1 previous similar message [ 3458.149318] Lustre: server umount lustre-OST0000 complete [ 3462.625413] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3462.637884] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3462.643084] Lustre: Skipped 1 previous similar message [ 3462.664188] Lustre: Skipped 1 previous similar message [ 3467.758173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3467.771226] Lustre: Skipped 1 previous similar message [ 3468.999986] Lustre: server umount lustre-MDT0000 complete [ 3472.867213] LustreError: 111134:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3473.319782] LustreError: 111115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787869030 with bad export cookie 14427272915914308010 [ 3473.321429] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3473.332523] LustreError: 111115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3473.778102] Lustre: server umount lustre-MDT0001 complete [ 3485.256892] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 3488.412701] Key type lgssc unregistered [ 3488.751804] LNet: 114779:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3488.774409] LNetError: 114779:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3488.801738] LNet: Removed LNI 192.168.204.126@tcp [ 3490.028619] Key type .llcrypt unregistered [ 3490.033238] Key type ._llcrypt unregistered [ 3508.461246] Key type ._llcrypt registered [ 3508.463796] Key type .llcrypt registered [ 3508.594879] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 3509.774442] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3509.797831] alg: No test for adler32 (adler32-zlib) [ 3510.806042] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [ 3511.015120] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 3512.656162] Key type lgssc registered [ 3513.917889] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3522.763489] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3522.805247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3524.331058] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3529.329229] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3534.825970] LustreError: 116000:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3534.841886] LustreError: 116000:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3538.722847] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3539.187802] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3543.900303] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3551.924882] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3558.431476] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3565.883658] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3566.229388] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3569.333241] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3573.162705] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3581.897392] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3594.725622] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3594.735886] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3594.766211] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3596.257799] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3596.264677] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3596.266966] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3596.284279] Lustre: Skipped 1 previous similar message [ 3600.489858] Lustre: server umount lustre-OST0000 complete [ 3605.995323] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3606.018310] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3606.042813] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3611.620198] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3611.622832] Lustre: Skipped 3 previous similar messages [ 3616.738282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3616.759017] Lustre: Skipped 1 previous similar message [ 3619.296107] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3619.518663] Lustre: server umount lustre-MDT0000 complete [ 3621.866688] LustreError: 115999:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3621.896882] LustreError: 115999:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3624.078699] LustreError: 116733:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787869181 with bad export cookie 9935170748524895035 [ 3624.089766] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3624.108773] LustreError: 116733:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3624.533431] Lustre: server umount lustre-MDT0001 complete [ 3636.737879] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 3639.761108] Key type lgssc unregistered [ 3640.093608] LNet: 119552:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3640.100292] LNetError: 119552:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3641.126889] LNet: Removed LNI 192.168.204.126@tcp [ 3642.340509] Key type .llcrypt unregistered [ 3642.342560] Key type ._llcrypt unregistered [ 3668.540221] Key type ._llcrypt registered [ 3668.545164] Key type .llcrypt registered [ 3668.691896] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 3670.002578] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3670.094347] alg: No test for adler32 (adler32-zlib) [ 3671.219688] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [ 3671.525615] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 3673.208176] Key type lgssc registered [ 3674.438735] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3683.700656] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3683.740991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3685.393041] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3691.191371] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3695.586349] LustreError: 120973:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3695.617937] LustreError: 120973:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3700.709800] LustreError: 120972:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3701.008512] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3701.531755] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3706.933364] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3714.868681] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3721.237249] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3730.773228] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3731.445960] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3740.051894] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3748.077404] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3756.041691] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 3765.239436] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3769.699070] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3769.713465] Lustre: Skipped 2 previous similar messages [ 3771.001068] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3771.066911] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3773.904551] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3773.908543] Lustre: Skipped 2 previous similar messages [ 3776.385634] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3776.390613] Lustre: Skipped 1 previous similar message [ 3784.557397] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.204.26@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3784.566927] Lustre: Skipped 2 previous similar messages [ 3785.364641] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.26@tcp 192.168.0.[2,10]@tcp [ 3789.297952] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.204.26/24@tcp in log lustre-MDT0000 [ 3789.306150] Lustre: Skipped 2 previous similar messages [ 3793.283544] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.26/24@tcp [ 3793.287032] Lustre: Skipped 1 previous similar message [ 3802.084403] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3802.086759] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3802.093690] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3802.101294] Lustre: Skipped 1 previous similar message [ 3806.635353] Lustre: server umount lustre-OST0000 complete [ 3812.322388] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3812.340528] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3812.345721] Lustre: Skipped 2 previous similar messages [ 3817.449961] LustreError: 120973:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3817.469501] LustreError: 120973:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3817.603507] Lustre: server umount lustre-MDT0000 complete [ 3821.830805] LustreError: 121707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787869378 with bad export cookie 8628045612423268674 [ 3821.836259] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3821.840531] LustreError: 121707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3822.346660] Lustre: server umount lustre-MDT0001 complete [ 3834.135760] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 3837.051288] Key type lgssc unregistered [ 3837.360945] LNet: 125706:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3837.369784] LNetError: 125706:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3837.384651] LNet: Removed LNI 192.168.204.126@tcp [ 3838.182312] Key type .llcrypt unregistered [ 3838.184474] Key type ._llcrypt unregistered [ 3874.535584] Key type ._llcrypt registered [ 3874.539718] Key type .llcrypt registered [ 3874.766590] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 3875.810123] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3876.037172] alg: No test for adler32 (adler32-zlib) [ 3877.085610] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [ 3877.242316] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 3878.912170] Key type lgssc registered [ 3880.339234] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3890.252587] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3890.275314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3891.900700] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3891.935876] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.204.26/24@tcp [ 3891.963062] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3896.626596] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3901.417733] LustreError: 127277:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3901.437913] LustreError: 127277:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3905.278686] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3905.577313] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3905.598696] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.26/24@tcp [ 3905.663432] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3910.575771] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3918.703360] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3925.718691] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3933.247681] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3933.669183] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3936.773766] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:201 to 0x280000400:225) [ 3941.601133] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3949.852333] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3962.340860] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3962.349920] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3962.365671] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3963.364272] Lustre: server umount lustre-OST0000 complete [ 3970.538549] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3970.545755] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3970.563974] Lustre: Skipped 1 previous similar message [ 3970.590550] Lustre: Skipped 1 previous similar message [ 3975.650255] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3975.661434] Lustre: Skipped 1 previous similar message [ 3980.772973] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3980.785118] Lustre: Skipped 1 previous similar message [ 3981.728256] Lustre: 126588:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787869522/real 1787869522] req@00000000b8a4a1ff x1874716979657216/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787869538 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3981.767283] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3982.304262] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3982.480043] Lustre: server umount lustre-MDT0000 complete [ 3986.913589] Lustre: 126587:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787869527/real 1787869527] req@00000000a444d328 x1874716979659008/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787869543 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3986.917076] LustreError: 128288:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3986.970790] Lustre: 126587:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3987.018210] LustreError: 128288:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3987.300302] LustreError: 127255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787869544 with bad export cookie 15798826487815596313 [ 3987.319752] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3987.323449] LustreError: 127255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3987.726626] Lustre: server umount lustre-MDT0001 complete [ 4000.654278] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 4004.032462] Key type lgssc unregistered [ 4004.381769] LNet: 130868:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4004.390065] LNetError: 130868:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4004.427089] LNet: Removed LNI 192.168.204.126@tcp [ 4005.511342] Key type .llcrypt unregistered [ 4005.515923] Key type ._llcrypt unregistered [ 4041.934553] Key type ._llcrypt registered [ 4041.936506] Key type .llcrypt registered [ 4042.092610] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 4060.135793] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 4061.860237] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4061.911771] alg: No test for adler32 (adler32-zlib) [ 4063.020925] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [ 4063.312611] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 4065.003176] Key type lgssc registered [ 4066.346349] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4110.279682] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4110.313342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4111.547205] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4111.576785] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4111.667833] Lustre: lustre-MDT0000: new disk, initializing [ 4111.768735] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4111.793580] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4116.999793] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4132.182116] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4132.401166] Lustre: 134271:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4132.488760] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4132.503616] Lustre: Skipped 1 previous similar message [ 4132.680623] Lustre: lustre-MDT0001: new disk, initializing [ 4132.842995] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4132.943151] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4132.972339] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4139.026965] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4151.462849] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4158.323357] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4167.836750] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4168.169544] Lustre: lustre-OST0000: new disk, initializing [ 4168.173722] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4168.185871] Lustre: 136243:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4168.306352] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4175.695315] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4177.519670] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4177.536786] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4177.642856] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4187.698281] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4193.335361] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4193.554386] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4198.859540] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4199.167986] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4203.500052] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4203.500152] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4203.512596] LustreError: Skipped 1 previous similar message [ 4203.513665] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4203.547313] Lustre: Skipped 1 previous similar message [ 4207.429231] Lustre: server umount lustre-OST0000 complete [ 4213.730164] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4213.735947] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4213.765866] Lustre: Skipped 1 previous similar message [ 4213.792162] Lustre: Skipped 2 previous similar messages [ 4218.216682] Lustre: server umount lustre-MDT0000 complete [ 4222.676261] LustreError: 134265:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787869779 with bad export cookie 493010274876924654 [ 4222.681720] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4222.696686] LustreError: 134265:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4223.246684] Lustre: server umount lustre-MDT0001 complete [ 4233.770892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4234.330269] LustreError: 138462:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4234.355229] LustreError: 138462:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 4234.413556] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4239.854828] LustreError: 138463:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4240.180761] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4244.965147] LustreError: 138462:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4250.089711] LustreError: 138463:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4251.300858] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4251.783127] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4257.802739] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4266.945316] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4274.742521] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4283.063229] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4283.562922] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4290.688617] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4300.005883] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4307.145683] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4307.404485] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4312.955895] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4313.153822] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4323.457303] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4323.713390] Lustre: lustre-OST0001: new disk, initializing [ 4323.719443] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4323.727533] Lustre: 141837:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4323.860850] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4331.149911] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4331.605403] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4331.611587] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4331.690408] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4339.565249] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4339.915925] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4344.842141] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4345.051571] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4352.881410] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9be3517ab800.ost_server_uuid 50 [ 4354.517642] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9be3517ab800.ost_server_uuid in IDLE state after 0 sec [ 4367.848875] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4367.849676] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4367.859293] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4367.862475] LustreError: Skipped 1 previous similar message [ 4371.678550] Lustre: server umount lustre-OST0001 complete [ 4372.966591] LustreError: 140285:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4372.979552] LustreError: 140285:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 4376.033904] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4376.043984] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4376.058642] Lustre: Skipped 1 previous similar message [ 4376.075358] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4376.083807] Lustre: Skipped 1 previous similar message [ 4383.217980] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4383.224130] LustreError: 141656:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4383.231150] Lustre: Skipped 3 previous similar messages [ 4383.267346] LustreError: 141656:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 4389.344166] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4389.654210] Lustre: server umount lustre-OST0000 complete [ 4394.979079] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4394.985035] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4394.999806] Lustre: Skipped 1 previous similar message [ 4395.007800] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4395.015134] Lustre: Skipped 2 previous similar messages [ 4408.288102] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4408.540404] Lustre: server umount lustre-MDT0000 complete [ 4408.803495] LustreError: 138463:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4408.818119] LustreError: 138463:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 4411.979314] LustreError: 141836:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787869968 with bad export cookie 493010274876925536 [ 4411.980826] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4411.983896] LustreError: 141836:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4412.243577] Lustre: server umount lustre-MDT0001 complete [ 4432.580787] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 4435.585996] Key type lgssc unregistered [ 4435.932354] LNet: 144848:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4435.943601] LNetError: 144848:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4435.962382] LNet: Removed LNI 192.168.204.126@tcp [ 4436.900320] Key type .llcrypt unregistered [ 4436.902885] Key type ._llcrypt unregistered [ 4483.598638] Key type ._llcrypt registered [ 4483.601544] Key type .llcrypt registered [ 4483.766710] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 4500.631180] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 4502.323747] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4502.421717] alg: No test for adler32 (adler32-zlib) [ 4503.826041] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [ 4504.221050] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 4505.912168] Key type lgssc registered [ 4507.759257] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4553.044677] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4553.091556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4554.682750] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4554.740820] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4554.864811] Lustre: lustre-MDT0000: new disk, initializing [ 4555.020848] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4555.054810] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4560.739900] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4573.676195] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4573.940378] Lustre: 149007:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4574.106964] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4574.124953] Lustre: Skipped 1 previous similar message [ 4574.301267] Lustre: lustre-MDT0001: new disk, initializing [ 4574.447821] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4574.535476] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4574.555872] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4581.202645] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4594.319593] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4600.177337] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4608.613522] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4609.076345] Lustre: lustre-OST0000: new disk, initializing [ 4609.093391] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4609.106949] Lustre: 150979:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4609.306421] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4617.312098] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4617.322219] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4617.489781] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4617.501436] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4628.555877] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4633.291870] Lustre: Failing over lustre-OST0000 [ 4633.666175] Lustre: server umount lustre-OST0000 complete [ 4651.496452] Lustre: 147228:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787870192/real 1787870192] req@000000004f53cf50 x1874717637119488/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1787870208 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4651.534349] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4653.977801] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4654.418077] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4654.471224] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4654.491677] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4654.608113] Lustre: 147228:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787870195/real 1787870195] req@0000000076465f47 x1874717637120896/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787870211 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4656.484946] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4659.169478] Lustre: 147227:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787870200/real 1787870200] req@000000003e988789 x1874717637122048/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787870216 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4659.213056] Lustre: 147227:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 4662.271928] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4662.272523] Lustre: lustre-OST0000: Recovery over after 0:06, of 3 clients 3 recovered and 0 were evicted. [ 4663.711898] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4664.289891] Lustre: 147229:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787870205/real 1787870205] req@000000000f6a77af x1874717637123200/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787870221 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4664.316397] Lustre: 147229:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4668.384272] Lustre: 147228:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787870208/real 1787870208] req@0000000051635009 x1874717637123968/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787870224 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4668.420098] Lustre: 147228:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4668.690236] Lustre: Failing over lustre-MDT0000 [ 4669.059929] Lustre: server umount lustre-MDT0000 complete [ 4671.969728] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4671.993340] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4672.016041] Lustre: Skipped 1 previous similar message [ 4677.090900] LustreError: 149015:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4677.135370] LustreError: 149015:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 7 previous similar messages [ 4682.228273] LustreError: 149019:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4682.267567] LustreError: 149019:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 4687.340714] LustreError: 149018:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4687.362292] LustreError: 149018:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 4688.345100] Lustre: 147227:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787870228/real 1787870228] req@0000000044b902f3 x1874717637134848/t0(0) o400->MGC192.168.204.126@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787870244 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4688.385802] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4690.466517] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4692.926517] LustreError: 149013:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4692.951971] LustreError: 149013:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 4698.035579] LustreError: 151198:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4698.055837] LustreError: 151198:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 4698.595231] LustreError: 147226:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000098df9b3e x1874717637140352/t0(0) o250->MGC192.168.204.126@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4699.151905] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4699.246933] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4703.181486] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4704.244294] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4704.259424] Lustre: Skipped 1 previous similar message [ 4704.282772] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4704.356661] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4705.328792] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4718.561702] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4718.577285] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4718.603192] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4719.588689] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4724.739114] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4724.753254] Lustre: Skipped 2 previous similar messages [ 4729.843175] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4729.852706] Lustre: Skipped 1 previous similar message [ 4730.848144] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4731.075681] Lustre: server umount lustre-OST0000 complete [ 4740.068274] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4740.069823] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4740.080756] Lustre: Skipped 1 previous similar message [ 4740.082726] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4741.914302] Lustre: server umount lustre-MDT0000 complete [ 4745.185924] LustreError: 149018:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4745.204852] LustreError: 149018:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 4746.438649] LustreError: 148999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787870303 with bad export cookie 496688352233982574 [ 4746.448899] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4746.461107] LustreError: 148999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4746.879360] Lustre: server umount lustre-MDT0001 complete [ 4760.155909] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 4763.521912] Key type lgssc unregistered [ 4763.850876] LNet: 154642:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4763.862599] LNetError: 154642:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4763.887753] LNet: Removed LNI 192.168.204.126@tcp [ 4764.832306] Key type .llcrypt unregistered [ 4764.836921] Key type ._llcrypt unregistered [ 4805.322223] Key type ._llcrypt registered [ 4805.326086] Key type .llcrypt registered [ 4805.463395] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 4807.236601] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4807.259544] alg: No test for adler32 (adler32-zlib) [ 4808.336209] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [ 4808.777991] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 4810.480369] Key type lgssc registered [ 4812.006559] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4821.330457] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4821.369509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4822.892829] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4828.718249] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4832.226782] LustreError: 156167:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4832.244338] LustreError: 156167:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 4837.346438] LustreError: 156168:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4840.540957] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4842.471221] LustreError: 156167:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4845.182213] LustreError: 156196:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4845.198124] LustreError: 156196:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4845.205600] LustreError: 156196:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 23, retries 0, failed: rc = -5 [ 4845.546244] Lustre: server umount lustre-MDT0000 complete [ 4862.620341] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 4874.352857] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 4917.029347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4917.368941] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4917.408804] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4917.527516] Lustre: lustre-MDT0000: new disk, initializing [ 4917.643591] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4917.659960] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4922.712470] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4934.560418] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4934.721980] Lustre: 159502:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4934.840353] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4934.843973] Lustre: Skipped 1 previous similar message [ 4934.937446] Lustre: lustre-MDT0001: new disk, initializing [ 4935.045545] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4935.107591] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4935.140018] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4940.159946] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4950.689454] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4958.278623] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4968.285452] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4968.823723] Lustre: lustre-OST0000: new disk, initializing [ 4968.833413] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4968.851587] Lustre: 161470:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4969.057885] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4976.481636] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4977.817538] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4977.828668] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4978.025071] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4988.689949] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4993.674980] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4996.059240] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 18:42:30 (1787870550) [ 5280.225464] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5280.238449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5280.246755] Lustre: Skipped 2 previous similar messages [ 5280.257296] Lustre: Skipped 1 previous similar message [ 5285.317186] Lustre: server umount lustre-MDT0000 complete [ 5285.346866] LustreError: 159509:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5285.365266] LustreError: 159509:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5289.438442] LustreError: 159494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787870846 with bad export cookie 15100598386831943878 [ 5289.440384] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5289.460351] LustreError: 159494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5290.467891] LustreError: 159510:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5290.470514] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5290.508221] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5290.510971] Lustre: Skipped 1 previous similar message [ 5294.499641] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5295.995514] Lustre: server umount lustre-MDT0001 complete [ 5307.713345] Lustre: server umount lustre-OST0000 complete [ 5320.048240] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 5340.299803] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 5386.519929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5387.122731] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5387.231913] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5387.539324] Lustre: lustre-MDT0000: new disk, initializing [ 5387.753603] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5387.821966] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5393.297596] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5404.000029] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5404.162530] Lustre: 166173:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5404.229907] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5404.233452] Lustre: Skipped 1 previous similar message [ 5404.326408] Lustre: lustre-MDT0001: new disk, initializing [ 5404.452630] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5404.518166] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5404.529668] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5410.254357] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5420.912972] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5427.045864] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5434.829758] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5435.290331] Lustre: lustre-OST0000: new disk, initializing [ 5435.298248] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5435.305749] Lustre: 168143:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5435.414960] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5443.340406] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5444.742850] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5444.765869] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5444.884612] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5453.705861] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5458.264823] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5458.630276] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5464.755950] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5464.955609] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5470.700857] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5470.704803] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5470.726439] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5470.732306] LustreError: Skipped 1 previous similar message [ 5470.740908] Lustre: Skipped 1 previous similar message [ 5473.311885] Lustre: server umount lustre-OST0000 complete [ 5480.931109] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5480.938805] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5480.954883] Lustre: Skipped 2 previous similar messages [ 5484.629910] Lustre: server umount lustre-MDT0000 complete [ 5486.049180] LustreError: 166186:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5486.065197] LustreError: 166186:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5488.972226] LustreError: 166166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787871045 with bad export cookie 15100598386832102708 [ 5488.975250] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5488.981896] LustreError: 166166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5489.643835] Lustre: server umount lustre-MDT0001 complete [ 5500.238623] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 18:50:54 (1787871054) [ 5520.384423] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 5532.121420] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 5574.704400] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5574.970801] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5575.019843] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5575.134204] Lustre: lustre-MDT0000: new disk, initializing [ 5575.242839] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5575.276199] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5580.209338] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5592.527832] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5592.693936] Lustre: 172901:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5592.951555] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5598.634468] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5609.423458] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5615.761846] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5623.559477] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5623.992692] Lustre: lustre-OST0000: new disk, initializing [ 5623.995877] Lustre: Skipped 1 previous similar message [ 5624.000768] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5624.007381] Lustre: Skipped 2 previous similar messages [ 5624.018379] Lustre: 174879:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5630.069360] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5630.074776] Lustre: Skipped 1 previous similar message [ 5630.082648] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5630.196877] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5631.890641] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5642.664288] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5656.036676] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5656.047389] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5656.058942] Lustre: Skipped 1 previous similar message [ 5656.071927] Lustre: Skipped 1 previous similar message [ 5658.397085] Lustre: server umount lustre-OST0000 complete [ 5664.737553] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5664.756338] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5670.020969] Lustre: server umount lustre-MDT0000 complete [ 5671.395275] LustreError: 172913:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5671.435558] LustreError: 172913:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 5674.938770] LustreError: 172894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787871231 with bad export cookie 15100598386832103590 [ 5674.944981] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5674.953155] LustreError: 172894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5675.498535] Lustre: server umount lustre-MDT0001 complete [ 5686.642647] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 18:54:01 (1787871241) [ 5705.721918] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 5716.683165] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 5756.666353] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5756.998296] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5757.049503] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5757.162302] Lustre: lustre-MDT0000: new disk, initializing [ 5757.266349] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5757.271832] Lustre: Skipped 2 previous similar messages [ 5757.301224] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5762.086759] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5773.300386] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5773.417193] Lustre: 179205:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5773.758732] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5779.509654] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5790.380552] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5796.802419] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5803.918223] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5804.150690] Lustre: 181181:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5810.986801] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5811.295762] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5811.409852] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5819.968406] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5832.163541] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5832.164952] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5832.174195] LustreError: Skipped 1 previous similar message [ 5832.177398] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5832.195936] Lustre: Skipped 2 previous similar messages [ 5832.212963] Lustre: Skipped 5 previous similar messages [ 5833.778230] Lustre: server umount lustre-OST0000 complete [ 5840.362189] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5844.894939] Lustre: server umount lustre-MDT0000 complete [ 5847.521568] LustreError: 180384:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5847.562461] LustreError: 180384:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 5849.478704] LustreError: 179196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787871406 with bad export cookie 15100598386832104675 [ 5849.495117] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5849.962121] Lustre: server umount lustre-MDT0001 complete [ 5860.815557] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 5863.591553] Key type lgssc unregistered [ 5863.897859] LNet: 183041:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5863.904484] LNetError: 183041:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5863.936218] LNet: Removed LNI 192.168.204.126@tcp [ 5864.824550] Key type .llcrypt unregistered [ 5864.832905] Key type ._llcrypt unregistered [ 5888.152895] Key type ._llcrypt registered [ 5888.156356] Key type .llcrypt registered [ 5888.271706] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 5889.541844] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5889.595847] alg: No test for adler32 (adler32-zlib) [ 5890.658900] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [ 5890.886710] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 5892.536192] Key type lgssc registered [ 5893.715761] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5902.250903] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5902.263613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5903.823210] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5908.894919] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5913.063929] LustreError: 184459:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5913.092348] LustreError: 184459:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5918.186164] LustreError: 184458:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5919.131307] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5919.535318] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5924.713673] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5932.012282] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5938.107724] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5945.896832] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5946.321935] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5948.417733] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5953.390188] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5961.092697] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5984.226050] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5984.235153] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5984.249714] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5986.284940] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5986.292341] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5986.316432] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5988.860383] Lustre: server umount lustre-OST0000 complete [ 5996.514924] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5996.524549] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5996.526126] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5996.526134] Lustre: Skipped 1 previous similar message [ 5996.544796] Lustre: Skipped 1 previous similar message [ 5999.765786] Lustre: server umount lustre-MDT0000 complete [ 6001.633273] LustreError: 184459:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6001.651279] LustreError: 184459:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 6003.921940] LustreError: 184439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787871560 with bad export cookie 9148857343920726010 [ 6003.932815] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6004.443492] Lustre: server umount lustre-MDT0001 complete [ 6018.006181] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 6021.993420] Key type lgssc unregistered [ 6022.278444] LNet: 188021:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6022.282816] LNetError: 188021:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6022.302205] LNet: Removed LNI 192.168.204.126@tcp [ 6023.235592] Key type .llcrypt unregistered [ 6023.237295] Key type ._llcrypt unregistered [ 6048.464466] Key type ._llcrypt registered [ 6048.465857] Key type .llcrypt registered [ 6048.621922] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 6049.720926] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6049.880921] alg: No test for adler32 (adler32-zlib) [ 6050.988927] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [ 6051.250636] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 6052.920228] Key type lgssc registered [ 6054.213079] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6063.118863] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6063.139675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6064.716066] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6069.990124] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6074.862641] LustreError: 189439:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6074.885375] LustreError: 189439:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 6079.189412] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6079.736571] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6083.433555] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6091.841532] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6099.188574] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6107.862299] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6108.403248] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6109.451022] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6115.019919] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6122.750724] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6130.659908] Lustre: lustre-OST0000: Not available for connect from 192.168.204.26@tcp (stopping) [ 6135.279695] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6135.281515] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6135.309725] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6135.324220] Lustre: Skipped 1 previous similar message [ 6135.339721] Lustre: Skipped 1 previous similar message [ 6136.930547] Lustre: server umount lustre-OST0000 complete [ 6147.060855] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9be3431d9800.ost_server_uuid 50 [ 6149.324125] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6165.987324] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6166.002686] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6166.011681] Lustre: Skipped 1 previous similar message [ 6170.291462] Lustre: server umount lustre-MDT0000 complete [ 6171.109527] LustreError: 189433:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6171.145874] LustreError: 189433:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 6174.729862] LustreError: 189421:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787871731 with bad export cookie 5196646273126135123 [ 6174.737188] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6174.742048] LustreError: 189421:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6175.109913] Lustre: server umount lustre-MDT0001 complete [ 6185.235309] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 19:02:20 (1787871740) [ 6193.279485] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6193.704353] LustreError: 193366:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6193.769898] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6198.536793] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6199.271228] LustreError: 193367:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6203.385243] LustreError: 193366:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6207.859330] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6208.276460] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6213.389529] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6221.714539] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6229.351538] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6237.633780] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6238.192469] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6239.247953] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6246.626454] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6255.193695] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6262.412678] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6262.702464] Lustre: lustre-OST0001: new disk, initializing [ 6262.705556] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6262.710519] Lustre: 196206:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6262.843470] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6270.356319] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6273.130787] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6273.140274] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6273.267796] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6281.052280] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6285.293754] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6285.305877] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6285.315652] Lustre: Skipped 1 previous similar message [ 6285.319760] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6288.880139] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6290.763611] Lustre: server umount lustre-OST0000 complete [ 6292.404359] LustreError: 195189:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6298.812414] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6299.214371] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6304.694379] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6304.955327] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6309.348167] LustreError: 196672:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6309.380348] LustreError: 196672:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 10 previous similar messages [ 6319.586549] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6319.594685] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6319.605139] LustreError: Skipped 1 previous similar message [ 6319.611054] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6319.632868] Lustre: Skipped 1 previous similar message [ 6319.655063] Lustre: Skipped 4 previous similar messages [ 6325.641863] Lustre: server umount lustre-OST0001 complete [ 6329.827307] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6329.844859] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6329.848407] Lustre: Skipped 1 previous similar message [ 6329.864268] Lustre: Skipped 3 previous similar messages [ 6336.035845] Lustre: server umount lustre-MDT0000 complete [ 6340.236115] LustreError: 193347:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787871897 with bad export cookie 5196646273126136194 [ 6340.246640] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6340.263686] LustreError: 193347:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6340.707600] Lustre: server umount lustre-MDT0001 complete [ 6367.928777] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 19:05:23 (1787871923) [ 6375.457284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6375.754373] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6375.802837] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6376.067899] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6380.826650] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6389.397642] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6389.487953] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6389.535940] Lustre: 199645:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6395.204520] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6403.540754] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6410.386577] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6420.000842] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6420.259044] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6426.479748] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6429.115874] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6436.891580] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6445.123626] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6445.256981] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6445.637739] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6445.652034] Lustre: Skipped 2 previous similar messages [ 6452.635871] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6461.040612] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6467.078194] Lustre: lustre-OST0000: Not available for connect from 192.168.204.26@tcp (stopping) [ 6467.081307] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6467.092420] Lustre: Skipped 3 previous similar messages [ 6467.125041] Lustre: Skipped 1 previous similar message [ 6478.306925] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6478.457755] Lustre: server umount lustre-OST0000 complete [ 6482.362122] LustreError: 201498:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6482.392835] LustreError: 201498:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 9 previous similar messages [ 6501.832830] Lustre: server umount lustre-OST0001 complete [ 6507.489458] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6507.505603] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6507.516121] Lustre: Skipped 2 previous similar messages [ 6507.523527] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6507.527217] Lustre: Skipped 9 previous similar messages [ 6513.198385] Lustre: server umount lustre-MDT0000 complete [ 6519.058968] LustreError: 201518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787872076 with bad export cookie 5196646273126137601 [ 6519.067484] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6519.079078] LustreError: 201518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6519.743897] Lustre: server umount lustre-MDT0001 complete [ 6549.899091] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 19:08:24 (1787872104) [ 6569.350178] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 6581.156924] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 6620.592735] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6620.892936] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6620.937723] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6621.059219] Lustre: lustre-MDT0000: new disk, initializing [ 6621.188399] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6621.234524] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6626.085369] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6637.465479] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6637.608919] Lustre: 207749:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6637.719726] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6637.730582] Lustre: Skipped 1 previous similar message [ 6637.845265] Lustre: lustre-MDT0001: new disk, initializing [ 6638.021385] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6638.037536] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6643.204685] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6653.978085] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6660.272064] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6670.158940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6670.457719] Lustre: lustre-OST0000: new disk, initializing [ 6670.467070] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6670.473348] Lustre: 209719:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6676.574156] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6676.583547] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6676.732056] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6678.897323] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6690.612891] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6696.452347] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6696.722291] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6702.246959] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6702.506027] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6707.695809] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6707.697282] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6707.702384] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6707.702395] Lustre: Skipped 2 previous similar messages [ 6707.704659] LustreError: Skipped 1 previous similar message [ 6707.730881] Lustre: Skipped 2 previous similar messages [ 6711.132927] Lustre: server umount lustre-OST0000 complete [ 6721.555228] Lustre: server umount lustre-MDT0000 complete [ 6723.041410] LustreError: 207755:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6723.069666] LustreError: 207755:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 13 previous similar messages [ 6725.643890] LustreError: 207742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787872282 with bad export cookie 5196646273126139162 [ 6725.660161] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6725.666775] LustreError: 207742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6726.064689] Lustre: server umount lustre-MDT0001 complete [ 6735.552297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6741.505262] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6750.123502] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6755.646821] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6763.990513] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6769.925892] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6776.684938] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6783.559855] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6791.218392] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6796.351231] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6796.658260] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6801.620637] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6801.896044] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6805.987045] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6809.817649] Lustre: server umount lustre-OST0000 complete [ 6816.311840] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6820.769472] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6850.446230] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6857.851756] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6866.417907] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6867.775498] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6871.275914] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6871.303500] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6871.310636] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6876.643807] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6876.656282] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6876.667493] Lustre: Skipped 5 previous similar messages [ 6876.669431] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6876.689990] Lustre: Skipped 6 previous similar messages [ 6878.931651] Lustre: server umount lustre-OST0000 complete [ 6889.502740] Lustre: server umount lustre-MDT0000 complete [ 6892.011483] LustreError: 213777:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6892.058192] LustreError: 213777:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 6893.745060] LustreError: 216739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787872450 with bad export cookie 5196646273126140051 [ 6893.754322] LustreError: 216739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6893.757116] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6894.236923] Lustre: server umount lustre-MDT0001 complete [ 6902.625099] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 19:14:17 (1787872457) [ 6910.242041] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6910.958752] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6910.965357] Lustre: Skipped 6 previous similar messages [ 6916.992268] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6926.539387] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6931.667028] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6938.433223] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6944.108381] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6951.290257] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6957.835403] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6965.086797] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6971.018132] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6971.208259] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6976.404000] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6976.650752] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6983.716499] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6983.929301] Lustre: lustre-OST0001: new disk, initializing [ 6983.935906] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6983.945367] Lustre: 221276:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6989.405308] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6989.417636] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6989.518068] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6990.753276] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7002.444755] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7006.935545] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7007.163595] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7010.515054] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7010.661793] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7015.393298] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7015.398969] LustreError: Skipped 3 previous similar messages [ 7018.561527] Lustre: server umount lustre-OST0001 complete [ 7026.460180] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7026.848287] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7032.181245] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7032.433521] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7062.673243] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7068.154450] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7068.173730] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7068.181093] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7068.197256] Lustre: Skipped 2 previous similar messages [ 7069.642922] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7076.936697] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7078.306512] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7083.491328] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7086.569149] Lustre: server umount lustre-OST0001 complete [ 7118.304539] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7118.543259] Lustre: server umount lustre-OST0000 complete [ 7128.746656] Lustre: server umount lustre-MDT0000 complete [ 7133.048599] LustreError: 217886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787872690 with bad export cookie 5196646273126141017 [ 7133.049107] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7133.066638] LustreError: 217886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7139.687287] Lustre: server umount lustre-MDT0001 complete [ 7165.517449] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 19:18:40 (1787872720) [ 7172.693110] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7173.135703] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7173.192827] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7178.408775] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7188.135776] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7188.292602] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7188.346134] Lustre: 226516:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7193.446594] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7201.060605] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7208.011851] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7215.945275] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7216.148186] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7223.611097] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7233.500247] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7244.589271] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7244.703105] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7251.853098] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7260.431610] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7262.449251] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7268.062918] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7268.268547] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7273.435510] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7273.789842] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7279.685707] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9be35daad800.ost_server_uuid 50 [ 7281.846538] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9be35daad800.ost_server_uuid in IDLE state after 0 sec [ 7283.460185] Lustre: Permanently deactivating lustre-OST0001 [ 7283.473592] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7287.950372] Lustre: Permanently reactivating lustre-OST0001 [ 7287.977605] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7287.986785] Lustre: Skipped 2 previous similar messages [ 7298.532818] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7298.553191] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7298.553204] Lustre: Skipped 15 previous similar messages [ 7298.559694] Lustre: Skipped 11 previous similar messages [ 7298.564648] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7298.572322] LustreError: Skipped 1 previous similar message [ 7299.780750] Lustre: server umount lustre-OST0001 complete [ 7303.662272] LustreError: 229105:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7303.693218] LustreError: 229105:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 43 previous similar messages [ 7310.865206] Lustre: server umount lustre-OST0000 complete [ 7322.767494] Lustre: server umount lustre-MDT0000 complete [ 7328.497193] LustreError: 226509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787872885 with bad export cookie 5196646273126142305 [ 7328.502306] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7328.512100] LustreError: 226509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7328.965747] Lustre: server umount lustre-MDT0001 complete [ 7359.778714] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 19:21:54 (1787872914) [ 7368.784900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7369.206170] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7369.243651] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7369.255189] Lustre: Skipped 2 previous similar messages [ 7374.240857] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7384.998428] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7385.154150] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7385.229083] Lustre: 233218:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7392.611851] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7403.640166] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7411.642292] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7420.662877] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7420.969367] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7420.997551] Lustre: Permanently deactivating lustre-OST0000 [ 7421.012271] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7427.171581] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7429.400665] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7438.930848] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7447.557675] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7447.657175] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7447.973349] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7447.986356] Lustre: Skipped 11 previous similar messages [ 7453.304192] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7455.326632] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7464.346664] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7466.449672] Lustre: Permanently reactivating lustre-OST0000 [ 7466.486354] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7466.491287] Lustre: Skipped 2 previous similar messages [ 7472.772422] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7472.774797] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 45 seconds. I think it's dead, and I am evicting it. exp 000000006ae095b6, cur 1787873029 deadline 1787873014 last 1787872984 [ 7472.788872] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7472.829847] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7472.894116] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7485.433389] Lustre: server umount lustre-OST0001 complete [ 7504.356272] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7504.731485] Lustre: server umount lustre-OST0000 complete [ 7515.595233] Lustre: server umount lustre-MDT0000 complete [ 7520.275568] LustreError: 233213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787873077 with bad export cookie 5196646273126143978 [ 7520.293753] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7520.298647] LustreError: 233213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7520.836855] Lustre: server umount lustre-MDT0001 complete [ 7534.408957] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 7538.402984] Key type lgssc unregistered [ 7538.757532] LNet: 238107:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7538.767698] LNetError: 238107:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7538.785981] LNet: Removed LNI 192.168.204.126@tcp [ 7539.706792] Key type .llcrypt unregistered [ 7539.710914] Key type ._llcrypt unregistered [ 7564.093446] Key type ._llcrypt registered [ 7564.097306] Key type .llcrypt registered [ 7564.238420] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 7565.013479] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7565.121348] alg: No test for adler32 (adler32-zlib) [ 7566.198156] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [ 7566.523836] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 7568.272249] Key type lgssc registered [ 7569.534544] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7582.867887] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7582.886195] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7584.457590] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7590.126746] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7594.991791] LustreError: 239579:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7595.007805] LustreError: 239579:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 7600.099704] LustreError: 239580:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7601.607053] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7601.843797] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7601.888052] Lustre: 239567:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7601.900055] Lustre: Permanently deactivating lustre-MDT0001 [ 7601.922417] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7602.438894] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7608.589383] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7611.963755] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7619.126523] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7627.111454] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7634.836138] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7635.273232] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7644.501116] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7647.540513] LustreError: 241423:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7652.842429] LustreError: 241899:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7654.496788] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7654.904542] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7660.006505] LustreError: 241423:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7660.035188] LustreError: 241423:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 7662.548479] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7662.858615] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7668.225098] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7668.245451] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7671.337490] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7682.475223] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7685.937209] Lustre: Permanently reactivating lustre-MDT0001 [ 7685.977731] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7685.985459] Lustre: Skipped 2 previous similar messages [ 7691.782687] Lustre: lustre-MDT0001: Client 9dd57134-592e-42d0-af6a-39e3702fe305 (at 192.168.204.26@tcp) reconnecting [ 7693.445665] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7693.459938] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7693.471439] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7693.518839] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7699.543565] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7701.852925] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7709.113164] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7709.459285] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7716.693692] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7717.002450] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7720.398801] Lustre: Permanently deactivating lustre-MDT0001 [ 7720.422456] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7720.427530] Lustre: Skipped 2 previous similar messages [ 7729.840843] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7739.876289] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7739.892561] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7739.905759] Lustre: Skipped 3 previous similar messages [ 7739.918722] Lustre: Skipped 3 previous similar messages [ 7742.535956] Lustre: server umount lustre-MDT0000 complete [ 7745.000504] LustreError: 240339:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7748.223664] LustreError: 244070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787873305 with bad export cookie 15173904618992731158 [ 7748.232845] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7748.238467] LustreError: 244070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7748.777961] Lustre: server umount lustre-MDT0001 complete [ 7754.704421] Lustre: server umount lustre-OST0000 complete [ 7771.664982] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 19:28:45 (1787873325) [ 7785.679231] Lustre: server umount lustre-OST0001 complete [ 7793.318507] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 7806.343199] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 7855.160325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7855.658809] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7855.688563] Lustre: Skipped 2 previous similar messages [ 7855.740763] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7855.881902] Lustre: lustre-MDT0000: new disk, initializing [ 7856.001078] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7856.029746] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7862.264603] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7875.626152] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7875.824866] Lustre: 247849:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7875.917951] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7875.923366] Lustre: Skipped 1 previous similar message [ 7876.119918] Lustre: lustre-MDT0001: new disk, initializing [ 7876.293885] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7876.398217] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7876.419942] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7883.000516] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7895.038230] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7902.990609] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7912.404692] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7912.943438] Lustre: lustre-OST0000: new disk, initializing [ 7912.956444] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7912.975131] Lustre: 249822:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7913.151341] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7919.168587] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7919.181580] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7919.349312] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7922.118845] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7933.993836] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7937.057187] LustreError: 248975:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7947.113232] LustreError: 248975:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7947.180121] LustreError: 247858:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7947.863676] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7948.144599] Lustre: lustre-OST0001: new disk, initializing [ 7948.153784] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7948.162565] Lustre: lustre-OST0001: Not available for connect from 192.168.204.26@tcp (not set up) [ 7948.174561] Lustre: 250938:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7948.329908] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7954.708515] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7954.718559] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7954.944662] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7957.236910] LustreError: 247858:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7958.014875] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7972.387045] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7974.768921] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7980.525440] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7980.530922] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7980.552145] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7984.028208] Lustre: server umount lustre-OST0001 complete [ 7985.645870] LustreError: 249805:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7985.674170] LustreError: 249805:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 8006.120371] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8006.126779] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8006.159164] Lustre: Skipped 2 previous similar messages [ 8006.163337] Lustre: Skipped 1 previous similar message [ 8008.825826] Lustre: server umount lustre-OST0000 complete [ 8014.829244] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8014.853927] LustreError: Skipped 1 previous similar message [ 8014.867554] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8014.899096] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8014.917707] Lustre: Skipped 1 previous similar message [ 8019.358595] Lustre: server umount lustre-MDT0000 complete [ 8021.484651] LustreError: 248974:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8021.510080] LustreError: 248974:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 9 previous similar messages [ 8023.626022] LustreError: 247843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787873580 with bad export cookie 15173904618992733755 [ 8023.628807] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8023.634385] LustreError: 247843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8030.162528] Lustre: server umount lustre-MDT0001 complete [ 8047.732582] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 8051.666407] Key type lgssc unregistered [ 8052.103725] LNet: 252977:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8052.129660] LNetError: 252977:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8052.196308] LNet: Removed LNI 192.168.204.126@tcp [ 8053.179946] Key type .llcrypt unregistered [ 8053.187631] Key type ._llcrypt unregistered [ 8100.886880] Key type ._llcrypt registered [ 8100.913440] Key type .llcrypt registered [ 8101.152497] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 8102.724977] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8102.825533] alg: No test for adler32 (adler32-zlib) [ 8103.894562] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [ 8104.285647] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 8105.944355] Key type lgssc registered [ 8106.693848] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8115.267642] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8115.327263] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8116.794568] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8116.839799] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8117.056068] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8121.542780] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8132.598499] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8132.726860] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8132.780757] Lustre: 255184:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8133.042365] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8138.099834] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8146.659627] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8153.474557] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8161.924441] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8162.150120] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8162.686469] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8170.531213] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8170.736111] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8180.241964] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8237.541978] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8237.546829] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8237.551689] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8237.559193] Lustre: Skipped 1 previous similar message [ 8237.577365] Lustre: Skipped 1 previous similar message [ 8242.664239] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8242.672899] Lustre: Skipped 1 previous similar message [ 8243.177817] Lustre: server umount lustre-OST0000 complete [ 8248.311122] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8262.857463] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8263.226269] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8267.322848] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8267.333826] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8267.335801] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8267.358626] Lustre: Skipped 1 previous similar message [ 8270.347352] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8278.993167] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8323.563974] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8323.573127] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8323.584907] Lustre: Skipped 1 previous similar message [ 8326.465834] Lustre: server umount lustre-OST0000 complete [ 8332.770373] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8332.784835] LustreError: Skipped 1 previous similar message [ 8332.796752] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8332.815241] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8332.820060] Lustre: Skipped 1 previous similar message [ 8337.584367] Lustre: server umount lustre-MDT0000 complete [ 8342.351591] LustreError: 259861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787873899 with bad export cookie 14251555990719173181 [ 8342.356171] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8342.368772] LustreError: 259861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8342.889332] Lustre: server umount lustre-MDT0001 complete [ 8355.443265] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 8358.482625] Key type lgssc unregistered [ 8358.870870] LNet: 260616:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8358.879346] LNetError: 260616:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8358.896630] LNet: Removed LNI 192.168.204.126@tcp [ 8359.922867] Key type .llcrypt unregistered [ 8359.927737] Key type ._llcrypt unregistered [ 8387.499452] Key type ._llcrypt registered [ 8387.501607] Key type .llcrypt registered [ 8387.610585] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 8389.119080] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8389.172503] alg: No test for adler32 (adler32-zlib) [ 8390.575502] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [ 8390.881632] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 8392.528168] Key type lgssc registered [ 8393.720236] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8402.606102] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8402.635205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8404.206522] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8408.631819] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8414.691191] LustreError: 262044:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8414.702036] LustreError: 262044:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 8417.436898] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8417.760433] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8422.526316] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8429.886368] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8435.687314] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8443.044166] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8443.304111] LustreError: 263865:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8443.467254] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8449.527750] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8449.771428] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8457.591950] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8484.321344] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8484.332942] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8484.343675] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8485.347907] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8485.350850] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8485.359952] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8485.373232] Lustre: Skipped 1 previous similar message [ 8488.667620] Lustre: server umount lustre-OST0000 complete [ 8494.572455] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8494.585687] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8494.611075] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8498.869421] Lustre: server umount lustre-MDT0000 complete [ 8500.707724] LustreError: 262044:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8500.752250] LustreError: 262044:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 8503.097954] LustreError: 262026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787874060 with bad export cookie 13197183109549530166 [ 8503.107776] LustreError: 262026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8503.108853] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8503.587835] Lustre: server umount lustre-MDT0001 complete [ 8516.716381] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 8519.831625] Key type lgssc unregistered [ 8520.164880] LNet: 266354:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8520.178329] LNetError: 266354:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8520.201777] LNet: Removed LNI 192.168.204.126@tcp [ 8521.114199] Key type .llcrypt unregistered [ 8521.117822] Key type ._llcrypt unregistered [ 8540.406565] Key type ._llcrypt registered [ 8540.411444] Key type .llcrypt registered [ 8540.537682] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 8541.583183] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8541.717043] alg: No test for adler32 (adler32-zlib) [ 8542.809361] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [ 8543.100291] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 8544.864723] Key type lgssc registered [ 8545.968975] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8553.671587] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8553.696847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8555.219595] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8559.946314] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8565.746757] LustreError: 267578:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8565.765312] LustreError: 267578:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 8569.268433] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8569.589760] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8574.480360] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8582.945703] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8589.890122] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8599.511357] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8599.784592] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8600.854949] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8606.955229] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8615.677376] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8629.738465] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8629.745470] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8629.766397] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8629.778867] Lustre: Skipped 1 previous similar message [ 8633.629226] Lustre: server umount lustre-OST0000 complete [ 8639.978245] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8639.983544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8640.008316] Lustre: Skipped 2 previous similar messages [ 8640.019281] Lustre: Skipped 1 previous similar message [ 8644.415070] Lustre: server umount lustre-MDT0000 complete [ 8645.093166] LustreError: 269062:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8645.119216] LustreError: 269062:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 8648.341252] LustreError: 267558:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787874205 with bad export cookie 6718478284157748693 [ 8648.350217] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8648.762396] Lustre: server umount lustre-MDT0001 complete [ 8659.641991] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 8662.796197] Key type lgssc unregistered [ 8663.140056] LNet: 271262:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8663.146304] LNetError: 271262:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8663.210751] LNet: Removed LNI 192.168.204.126@tcp [ 8664.160693] Key type .llcrypt unregistered [ 8664.165088] Key type ._llcrypt unregistered [ 8690.198751] Key type ._llcrypt registered [ 8690.203965] Key type .llcrypt registered [ 8690.343247] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 8691.612559] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8691.633170] alg: No test for adler32 (adler32-zlib) [ 8692.933256] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [ 8693.201594] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 8694.888182] Key type lgssc registered [ 8696.305896] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8704.426347] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8704.460488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8705.961255] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8710.441916] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8715.239369] LustreError: 272679:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8715.278322] LustreError: 272679:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 8719.661914] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8720.044082] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8725.340383] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8734.150722] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8740.279275] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8748.107337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8748.670610] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8750.727220] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8756.403043] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8764.814693] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8796.131095] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8796.134348] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8796.155001] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8796.165259] Lustre: Skipped 1 previous similar message [ 8799.327660] Lustre: server umount lustre-OST0000 complete [ 8806.373291] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8806.377557] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8806.389458] Lustre: Skipped 1 previous similar message [ 8806.409193] Lustre: Skipped 2 previous similar messages [ 8810.462597] Lustre: server umount lustre-MDT0000 complete [ 8811.496181] LustreError: 272679:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8811.524041] LustreError: 272679:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 8815.675574] LustreError: 272661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787874372 with bad export cookie 8890385043247970905 [ 8815.693574] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8815.703032] LustreError: 272661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8816.662250] Lustre: server umount lustre-MDT0001 complete [ 8830.635461] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 8834.205444] Key type lgssc unregistered [ 8834.552268] LNet: 276990:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8834.564939] LNetError: 276990:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8834.583902] LNet: Removed LNI 192.168.204.126@tcp [ 8835.614379] Key type .llcrypt unregistered [ 8835.618560] Key type ._llcrypt unregistered [ 8855.170765] Key type ._llcrypt registered [ 8855.173549] Key type .llcrypt registered [ 8855.292647] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 8856.561701] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8856.598575] alg: No test for adler32 (adler32-zlib) [ 8857.675364] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [ 8858.053812] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 8859.776235] Key type lgssc registered [ 8861.056145] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8870.460963] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8870.476488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8872.156893] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8877.357189] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8882.676940] LustreError: 278229:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8882.709746] LustreError: 278229:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 8887.782650] LustreError: 278231:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8888.374303] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8888.830789] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8894.684445] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8904.252948] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8911.177771] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8918.917389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8919.277027] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8926.710675] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8934.436236] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8942.830392] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8956.137706] Lustre: server umount lustre-OST0000 complete [ 8960.481815] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8960.488065] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8960.504535] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8963.556519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8963.556634] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8968.679082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8968.695058] Lustre: Skipped 2 previous similar messages [ 8970.823875] Lustre: server umount lustre-MDT0000 complete [ 8973.795821] LustreError: 278232:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8973.846946] LustreError: 278232:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 8974.817127] Lustre: 277527:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787874515/real 1787874515] req@00000000e7cef10d x1874722202615936/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787874531 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8974.850671] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8975.023628] LustreError: 278196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787874532 with bad export cookie 18383342225683992045 [ 8975.032052] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8975.032973] LustreError: 278196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8975.586255] Lustre: server umount lustre-MDT0001 complete [ 8987.987162] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 8991.056538] Key type lgssc unregistered [ 8991.437883] LNet: 281955:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8991.446543] LNetError: 281955:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8991.467180] LNet: Removed LNI 192.168.204.126@tcp [ 8992.268297] Key type .llcrypt unregistered [ 8992.270807] Key type ._llcrypt unregistered [ 9005.569489] Key type ._llcrypt registered [ 9005.575892] Key type .llcrypt registered [ 9005.780788] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 9025.055621] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 9042.643694] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 9043.707347] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9043.781329] alg: No test for adler32 (adler32-zlib) [ 9045.017802] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [ 9045.281195] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 9046.934270] Key type lgssc registered [ 9047.983680] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9091.871061] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9091.940691] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9093.367038] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9093.419943] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9093.564604] Lustre: lustre-MDT0000: new disk, initializing [ 9093.662356] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9093.688374] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9099.323317] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9112.165462] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9112.384524] Lustre: 285509:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9112.470916] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9112.479948] Lustre: Skipped 1 previous similar message [ 9112.586462] Lustre: lustre-MDT0001: new disk, initializing [ 9112.723943] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9112.800938] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9112.813672] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9118.019604] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9130.037509] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9136.974362] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9145.249453] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9145.807078] Lustre: lustre-OST0000: new disk, initializing [ 9145.812657] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9145.825506] Lustre: 287482:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9145.949423] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9151.609451] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9151.624356] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9151.830511] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9153.927895] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9165.834869] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9171.497106] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9171.807591] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9178.241561] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9178.526113] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9182.689778] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9182.700667] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9182.735212] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9182.739551] Lustre: Skipped 1 previous similar message [ 9187.337608] Lustre: server umount lustre-OST0000 complete [ 9192.933741] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9192.936250] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9192.953813] Lustre: Skipped 2 previous similar messages [ 9192.964591] Lustre: Skipped 1 previous similar message [ 9198.057596] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9198.072050] Lustre: Skipped 1 previous similar message [ 9202.155091] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9202.170262] Lustre: Skipped 1 previous similar message [ 9206.240141] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9206.703933] Lustre: server umount lustre-MDT0000 complete [ 9211.790639] LustreError: 285500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787874768 with bad export cookie 16857467600522478878 [ 9211.796337] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9211.807021] LustreError: 285500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9212.319972] Lustre: server umount lustre-MDT0001 complete [ 9222.908206] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 19:52:57 (1787874777) [ 9230.718684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9231.239709] LustreError: 289897:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9231.255278] LustreError: 289897:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 9231.359454] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9236.449922] LustreError: 289898:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9238.518226] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9241.574589] LustreError: 289897:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9246.696881] LustreError: 289898:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9249.020544] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9249.616466] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9255.015464] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9264.249689] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9271.045514] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9278.012189] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9278.528832] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9286.183959] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9295.485622] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9314.133727] Lustre: server umount lustre-OST0000 complete [ 9321.476928] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9321.488412] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9321.488907] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9321.488912] LustreError: Skipped 1 previous similar message [ 9321.504114] Lustre: Skipped 1 previous similar message [ 9328.688528] Lustre: server umount lustre-MDT0000 complete [ 9331.681422] LustreError: 290868:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9331.699828] LustreError: 290868:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 9332.704386] Lustre: 283733:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787874873/real 1787874873] req@00000000559f5cfe x1874722398747776/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787874889 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9332.705862] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9332.734533] Lustre: 283733:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 9332.901302] LustreError: 289879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787874889 with bad export cookie 16857467600522479760 [ 9332.907731] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9332.916452] LustreError: 289879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9333.485270] Lustre: server umount lustre-MDT0001 complete [ 9346.263559] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 9349.793986] Key type lgssc unregistered [ 9350.128202] LNet: 293460:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9350.137931] LNetError: 293460:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9350.153608] LNet: Removed LNI 192.168.204.126@tcp [ 9351.052475] Key type .llcrypt unregistered [ 9351.056433] Key type ._llcrypt unregistered [ 9379.889884] Key type ._llcrypt registered [ 9379.893324] Key type .llcrypt registered [ 9379.995900] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 9393.780682] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 9395.220047] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9395.254690] alg: No test for adler32 (adler32-zlib) [ 9396.479945] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [ 9396.818613] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 9398.536970] Key type lgssc registered [ 9399.704969] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9445.722991] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9445.748457] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9446.944643] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9447.004827] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9447.147282] Lustre: lustre-MDT0000: new disk, initializing [ 9447.224264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9447.248717] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9451.729366] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9462.127938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9462.278829] Lustre: 297285:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9462.341050] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9462.344244] Lustre: Skipped 1 previous similar message [ 9462.453782] Lustre: lustre-MDT0001: new disk, initializing [ 9462.537449] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9462.573606] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9462.583590] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9467.311512] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9477.748972] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9483.841287] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9491.288977] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9491.485726] Lustre: lustre-OST2710: new disk, initializing [ 9491.488540] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9491.493116] Lustre: 299253:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9491.574858] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9497.476523] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9501.858816] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9501.868645] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9501.983712] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9507.233047] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9514.323653] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9514.523841] Lustre: lustre-OST03e8: new disk, initializing [ 9514.526790] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9514.532249] Lustre: 300316:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9514.622381] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9520.260571] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9520.268937] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9520.416914] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9521.013205] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9529.122634] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9537.304418] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9537.521847] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9542.267126] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9542.487326] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9547.486563] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9547.747173] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9552.528600] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9552.801950] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9559.513600] Lustre: server umount lustre-MDT0000 complete [ 9560.033544] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9560.047641] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9561.583719] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9561.599485] Lustre: Skipped 2 previous similar messages [ 9563.999606] LustreError: 299252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787875120 with bad export cookie 7841669394670906687 [ 9564.011929] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9564.020317] LustreError: 299252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9564.691982] Lustre: server umount lustre-MDT0001 complete [ 9578.287189] Lustre: server umount lustre-OST2710 complete [ 9592.673972] Lustre: server umount lustre-OST03e8 complete [ 9612.779522] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 9624.236519] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 9659.178504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9659.342264] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9659.382891] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9659.480885] Lustre: lustre-MDT0000: new disk, initializing [ 9659.626901] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9659.660047] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9665.328726] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9676.888332] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9677.111263] Lustre: 305571:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9677.173551] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9677.175932] Lustre: Skipped 1 previous similar message [ 9677.223163] Lustre: lustre-MDT0001: new disk, initializing [ 9677.290492] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9677.314774] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9677.321633] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9681.105969] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9688.461734] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9694.045122] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9706.101326] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9706.891881] Lustre: lustre-OST0000: new disk, initializing [ 9706.907606] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9706.919687] Lustre: 307549:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9707.028526] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9714.463905] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9714.489691] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9714.805840] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9721.917831] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9746.334279] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9756.167509] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9756.465637] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9763.909982] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9764.266680] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9770.982524] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9770.988255] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9771.016639] Lustre: Skipped 1 previous similar message [ 9771.019427] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9771.066839] Lustre: Skipped 1 previous similar message [ 9773.645770] Lustre: server umount lustre-OST0000 complete [ 9781.217869] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9781.238617] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9781.258970] Lustre: Skipped 1 previous similar message [ 9786.343870] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9786.353451] Lustre: Skipped 1 previous similar message [ 9786.917642] Lustre: server umount lustre-MDT0000 complete [ 9791.036490] LustreError: 305561:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787875348 with bad export cookie 7841669394670908374 [ 9791.039019] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9791.058942] LustreError: 305561:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9791.455396] Lustre: server umount lustre-MDT0001 complete [ 9800.832147] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 20:02:35 (1787875355) [ 9802.106279] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9804.259869] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 20:02:38 (1787875358) [ 9807.758677] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 9835.409212] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9835.762615] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9835.777870] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9835.876611] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9841.531888] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9850.934423] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9851.088592] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9851.128177] Lustre: 311413:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9856.711442] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9879.088713] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9879.463152] Lustre: Denying initial registration attempt from nid 192.168.204.126@tcp, specified as failover [ 9879.471223] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9879.485112] LustreError: 312869:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 9879.497187] LustreError: 312869:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9879.502858] LustreError: 312869:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9879.589903] Lustre: server umount lustre-OST0000 complete [ 9879.593801] LustreError: 312869:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9887.206800] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9887.214135] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9887.228032] Lustre: Skipped 1 previous similar message [ 9887.239928] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9891.307515] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9891.321137] Lustre: Skipped 1 previous similar message [ 9892.901460] Lustre: server umount lustre-MDT0000 complete [ 9896.418253] LustreError: 312364:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9896.452772] LustreError: 312364:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 7 previous similar messages [ 9896.857412] LustreError: 312164:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787875453 with bad export cookie 7841669394670909256 [ 9896.867891] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9896.873094] LustreError: 312164:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9897.421720] Lustre: server umount lustre-MDT0001 complete [ 9909.452882] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 9913.210184] Key type lgssc unregistered [ 9913.573564] LNet: 313993:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9913.596307] LNetError: 313993:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9913.626882] LNet: Removed LNI 192.168.204.126@tcp [ 9914.581700] Key type .llcrypt unregistered [ 9914.584085] Key type ._llcrypt unregistered [ 9927.279723] Key type ._llcrypt registered [ 9927.281797] Key type .llcrypt registered [ 9927.377904] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 9928.405414] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9928.458354] alg: No test for adler32 (adler32-zlib) [ 9929.563672] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [ 9929.819852] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 9931.464249] Key type lgssc registered [ 9932.941517] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9984.917786] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 9994.309793] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9994.342823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9995.655972] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9995.691506] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9995.995217] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10001.095560] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10010.885423] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10011.007202] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10011.042975] Lustre: 316759:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10011.522381] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10016.837434] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10026.090190] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10032.938697] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10053.416100] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10053.570348] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10053.780587] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10060.965467] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10068.933447] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10071.534050] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10071.536321] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10071.538557] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10071.542124] Lustre: Skipped 1 previous similar message [10071.558201] LustreError: Skipped 1 previous similar message [10076.644385] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10076.658987] Lustre: Skipped 2 previous similar messages [10081.773335] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10081.778538] Lustre: Skipped 1 previous similar message [10085.344122] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10085.444074] Lustre: server umount lustre-OST0000 complete [10092.003940] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10092.009689] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10092.022348] Lustre: Skipped 1 previous similar message [10092.037104] Lustre: Skipped 1 previous similar message [10095.887181] Lustre: server umount lustre-MDT0000 complete [10099.797234] LustreError: 317499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787875656 with bad export cookie 16272822357123576361 [10099.811795] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10099.812306] LustreError: 317499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10100.378220] Lustre: server umount lustre-MDT0001 complete [10111.567038] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [10113.971844] Key type lgssc unregistered [10114.182314] LNet: 320338:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10114.196733] LNetError: 320338:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10114.216776] LNet: Removed LNI 192.168.204.126@tcp [10114.787510] Key type .llcrypt unregistered [10114.789916] Key type ._llcrypt unregistered [10133.386408] Key type ._llcrypt registered [10133.388262] Key type .llcrypt registered [10133.482889] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [10134.172295] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10134.233017] alg: No test for adler32 (adler32-zlib) [10135.279603] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [10135.485620] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [10137.184296] Key type lgssc registered [10138.306962] Lustre: Echo OBD driver; http://www.lustre.org/ [10145.589496] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10145.620445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10146.992721] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10151.191728] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10156.520930] LustreError: 321732:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10156.533365] LustreError: 321732:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [10159.429147] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10159.786672] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10164.863508] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10171.993385] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10177.112870] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10184.368976] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10184.664289] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10189.460630] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10196.470752] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10206.176958] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10206.191810] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10206.214378] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10207.208403] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10207.213419] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10209.750724] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.26@tcp (stopping) [10209.756726] Lustre: Skipped 2 previous similar messages [10211.997677] Lustre: server umount lustre-MDT0000 complete [10212.325102] LustreError: 322974:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10212.340071] LustreError: 322974:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [10214.834170] LustreError: 321728:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10215.871995] LustreError: 323577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787875772 with bad export cookie 9089386306593488884 [10215.886012] LustreError: 323577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10215.886715] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10217.442521] LustreError: 324033:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10217.444501] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10217.472795] Lustre: Skipped 1 previous similar message [10217.479379] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10222.381625] Lustre: server umount lustre-MDT0001 complete [10229.673879] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10238.431497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10240.544199] LustreError: 325313:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [10240.551945] LustreError: 325313:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@00000000913c7734 x1874723541645952/t0(0) o250->MGC192.168.204.126@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1787875797 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10240.579745] LustreError: 325313:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [10241.308676] LustreError: 323565:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10241.321328] LustreError: 323565:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [10241.397932] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10241.469508] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [10245.206911] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10250.598222] LustreError: 325327:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10250.599395] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10250.609072] LustreError: 325327:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [10252.611293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10252.912437] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10256.572782] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10258.406460] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10262.047368] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10263.098909] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10267.065862] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10268.331415] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10272.226542] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10272.235159] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10272.255922] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10272.258585] Lustre: Skipped 2 previous similar messages [10273.253164] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10277.724383] Lustre: server umount lustre-OST0000 complete [10283.489480] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10283.498546] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10283.509838] Lustre: Skipped 1 previous similar message [10283.514080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10283.517601] Lustre: Skipped 2 previous similar messages [10295.264163] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10295.399200] Lustre: server umount lustre-MDT0000 complete [10298.340842] LustreError: 325330:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10298.353567] LustreError: 325330:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [10298.496456] LustreError: 321713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787875855 with bad export cookie 9089386306593496024 [10298.500477] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10298.503592] LustreError: 321713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10298.809764] Lustre: server umount lustre-MDT0001 complete [10307.564532] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [10309.940254] Key type lgssc unregistered [10310.190597] LNet: 328002:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10310.197832] LNetError: 328002:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10310.215082] LNet: Removed LNI 192.168.204.126@tcp [10310.907924] Key type .llcrypt unregistered [10310.911397] Key type ._llcrypt unregistered [10331.942341] Key type ._llcrypt registered [10331.946342] Key type .llcrypt registered [10332.047417] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [10333.256994] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10333.292685] alg: No test for adler32 (adler32-zlib) [10334.323915] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [10334.549453] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [10336.224167] Key type lgssc registered [10337.379765] Lustre: Echo OBD driver; http://www.lustre.org/ [10344.068044] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10344.087409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10345.413544] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10349.244688] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10355.681824] LustreError: 329414:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10355.708701] LustreError: 329414:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [10357.189436] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10357.490565] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10362.076439] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10371.045861] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10379.539351] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10387.427868] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10387.433802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10387.460718] Lustre: Skipped 1 previous similar message [10387.483301] Lustre: Skipped 1 previous similar message [10390.450368] Lustre: server umount lustre-MDT0000 complete [10392.557987] LustreError: 329411:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10392.578163] LustreError: 329411:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [10396.216777] LustreError: 331031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787875953 with bad export cookie 11730464767613769614 [10396.226230] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10396.259395] LustreError: 331031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10397.065483] Lustre: server umount lustre-MDT0001 complete [10417.498922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10417.740616] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10417.764613] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10418.026758] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10423.634162] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10433.455234] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10433.528874] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10433.560782] Lustre: 332004:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10433.919413] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10438.406599] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10448.831763] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10459.553816] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10473.520602] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10473.855057] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10473.909495] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10473.924788] LustreError: 333888:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [10473.944164] LustreError: 333888:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [10473.968787] LustreError: 333888:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10474.150894] Lustre: server umount lustre-OST0000 complete [10474.156497] LustreError: 333888:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [10488.633825] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10488.875148] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10489.086110] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10494.654542] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10495.980734] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10503.823451] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10512.925816] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10513.056538] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10513.171304] Lustre: lustre-OST0001: new disk, initializing [10513.181722] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10513.196810] Lustre: 335431:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10513.386332] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10519.246342] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10519.282229] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10519.493812] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10522.416420] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10532.908973] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10540.001419] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10540.009456] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10540.032658] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10540.039324] Lustre: Skipped 1 previous similar message [10542.789988] Lustre: server umount lustre-OST0001 complete [10543.020593] LustreError: 334355:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10543.029443] LustreError: 334355:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [10548.201759] LustreError: 334355:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10548.205577] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10548.212164] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10548.217458] LustreError: 334355:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [10548.236880] Lustre: Skipped 2 previous similar messages [10552.606837] Lustre: server umount lustre-OST0000 complete [10556.905500] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10556.914593] LustreError: Skipped 1 previous similar message [10556.917509] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10556.925593] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10556.929612] Lustre: Skipped 1 previous similar message [10563.556567] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10563.568880] Lustre: Skipped 3 previous similar messages [10571.232159] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10571.436595] Lustre: server umount lustre-MDT0000 complete [10573.795269] LustreError: 333179:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10573.823943] LustreError: 333179:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [10575.733242] LustreError: 331996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787876132 with bad export cookie 11730464767613770160 [10575.739842] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10575.749253] LustreError: 331996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10576.124928] Lustre: server umount lustre-MDT0001 complete [10586.824737] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [10589.451981] Key type lgssc unregistered [10589.877782] LNet: 337395:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10589.887869] LNetError: 337395:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10590.954770] LNet: Removed LNI 192.168.204.126@tcp [10591.774119] Key type .llcrypt unregistered [10591.776350] Key type ._llcrypt unregistered [10670.074125] Key type ._llcrypt registered [10670.076022] Key type .llcrypt registered [10670.220358] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [10685.747826] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [10687.270933] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10687.404260] alg: No test for adler32 (adler32-zlib) [10689.046082] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [10689.404367] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [10691.096204] Key type lgssc registered [10692.728601] Lustre: Echo OBD driver; http://www.lustre.org/ [10735.228491] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10735.261144] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10736.645719] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10736.701191] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10736.796661] Lustre: lustre-MDT0000: new disk, initializing [10736.910249] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10736.932108] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10742.205598] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10754.178714] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10754.406281] Lustre: 342333:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10754.546390] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10754.561081] Lustre: Skipped 1 previous similar message [10754.650518] Lustre: lustre-MDT0001: new disk, initializing [10754.752657] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10754.807470] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10754.825546] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10759.606375] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10769.374520] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10775.791680] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10783.504859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10783.782851] Lustre: lustre-OST0000: new disk, initializing [10783.787186] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10783.793913] Lustre: 344304:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10783.878390] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10790.014882] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10790.027241] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10790.176601] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10790.564513] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10801.110593] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10806.332070] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10806.639482] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10811.394766] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10811.684880] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10815.971368] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10815.979675] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10816.009774] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10816.010673] LustreError: Skipped 1 previous similar message [10821.096779] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10821.112241] Lustre: Skipped 2 previous similar messages [10826.211592] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10826.227344] Lustre: Skipped 1 previous similar message [10828.258716] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10828.440843] Lustre: server umount lustre-OST0000 complete [10836.451863] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10836.456449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10836.461304] Lustre: Skipped 2 previous similar messages [10836.477797] Lustre: Skipped 1 previous similar message [10838.541711] Lustre: server umount lustre-MDT0000 complete [10841.770438] LustreError: 342327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787876398 with bad export cookie 17304419348969408053 [10841.771315] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10841.776328] LustreError: 342327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10842.056487] Lustre: server umount lustre-MDT0001 complete [10850.345832] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 20:20:05 (1787876405) [10858.126442] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 20:20:12 (1787876412) [10867.729267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10868.208613] LustreError: 347059:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10868.233799] LustreError: 347059:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [10868.320303] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10873.314450] LustreError: 347060:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10874.253377] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10878.433270] LustreError: 347059:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10883.553349] LustreError: 347060:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10884.405783] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10884.720396] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10888.878600] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10897.652668] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10904.326042] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10913.294383] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10913.754375] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10919.530898] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10929.054829] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10939.242749] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10941.281786] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10944.194618] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10946.142369] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10948.892114] Lustre: Failing over lustre-MDT0000 [10949.054669] Lustre: server umount lustre-MDT0000 complete [10949.603855] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10949.620169] LustreError: 347056:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10949.623768] Lustre: Skipped 2 previous similar messages [10949.656220] LustreError: 347056:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [10958.257941] LustreError: 347056:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10958.273423] LustreError: 347056:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [10965.986080] Lustre: 340559:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787876506/real 1787876506] req@00000000aa0502ae x1874724122600832/t0(0) o400->MGC192.168.204.126@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787876522 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10966.007543] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10968.216036] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10976.229781] LustreError: 340555:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000ed0886c9 x1874724122606208/t0(0) o250->MGC192.168.204.126@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10976.250996] LustreError: 347059:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10976.278407] LustreError: 347059:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 17 previous similar messages [10976.594674] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10976.665801] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10977.762991] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10981.903774] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10981.943481] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [10982.095573] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10983.818380] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10998.917887] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11000.986733] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11005.157927] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [11007.354779] Lustre: Failing over lustre-MDT0000 [11007.461742] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11007.480278] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11007.481457] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11007.509573] Lustre: Skipped 1 previous similar message [11007.532624] Lustre: Skipped 2 previous similar messages [11007.987773] Lustre: server umount lustre-MDT0000 complete [11008.464682] LustreError: 347055:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11028.768242] Lustre: 340558:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787876569/real 1787876569] req@00000000a1710c4d x1874724122624512/t0(0) o400->MGC192.168.204.126@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787876585 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11028.808330] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11031.045180] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11038.178930] LustreError: 340555:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000008c6462e2 x1874724122629632/t0(0) o250->MGC192.168.204.126@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11038.584402] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11038.655867] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11039.153601] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11043.832577] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11043.846675] Lustre: Skipped 2 previous similar messages [11043.882540] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [11043.956183] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11044.377863] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [11055.399281] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11057.060721] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11064.301234] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11064.310670] Lustre: Skipped 1 previous similar message [11064.313778] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11064.315624] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11070.726190] Lustre: server umount lustre-MDT0000 complete [11074.533532] LustreError: 347789:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11074.542322] LustreError: 347789:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 28 previous similar messages [11076.469476] LustreError: 347040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787876633 with bad export cookie 17304419348969410769 [11076.481527] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11076.494392] LustreError: 347040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11077.007670] Lustre: server umount lustre-MDT0001 complete [11091.353476] Lustre: server umount lustre-OST0000 complete [11109.957651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11110.884568] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11116.912901] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [11127.409642] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11132.706856] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [11140.454719] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11146.381817] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11153.612375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11155.260112] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11160.891605] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [11168.697959] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11181.025621] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11181.031256] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11181.040798] Lustre: Skipped 2 previous similar messages [11181.049123] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11181.067806] Lustre: Skipped 5 previous similar messages [11186.004642] Lustre: server umount lustre-OST0000 complete [11197.349244] Lustre: server umount lustre-MDT0000 complete [11202.039150] LustreError: 353392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787876759 with bad export cookie 17304419348969411504 [11202.052825] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11202.055260] LustreError: 353392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11202.575175] Lustre: server umount lustre-MDT0001 complete [11213.854286] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [11216.872730] Key type lgssc unregistered [11217.215696] LNet: 356946:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11217.228563] LNetError: 356946:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11217.254464] LNet: Removed LNI 192.168.204.126@tcp [11218.143447] Key type .llcrypt unregistered [11218.145399] Key type ._llcrypt unregistered [11244.544699] Key type ._llcrypt registered [11244.549139] Key type .llcrypt registered [11244.673645] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [11245.976219] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11246.028966] alg: No test for adler32 (adler32-zlib) [11247.256978] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [11247.545682] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [11249.241639] Key type lgssc registered [11250.388520] Lustre: Echo OBD driver; http://www.lustre.org/ [11259.422914] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11259.444379] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11260.969792] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11267.043654] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [11270.119709] LustreError: 358462:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11270.161305] LustreError: 358462:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [11275.243855] LustreError: 358463:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11276.240792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11276.704528] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11280.797593] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [11289.530528] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11297.550702] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11307.050552] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11307.566851] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11313.981900] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [11324.538781] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11330.109881] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11343.131889] Lustre: server umount lustre-MDT0000 complete [11343.336909] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11343.344375] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11343.373286] LustreError: 358463:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11345.378696] LustreError: 358462:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11345.391226] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11348.222598] LustreError: 359197:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787876905 with bad export cookie 10864596379666664370 [11348.225842] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11348.230926] LustreError: 359197:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11348.641834] Lustre: server umount lustre-MDT0001 complete [11364.759812] Lustre: server umount lustre-OST0000 complete [11378.802955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11379.184890] LustreError: 362197:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11379.204809] LustreError: 362197:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [11379.247070] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11384.219649] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [11389.411477] LustreError: 362197:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11389.441609] LustreError: 362197:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [11393.803365] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11394.231632] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11398.660235] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [11408.391807] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11415.434110] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11426.830755] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11427.547744] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11436.946071] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [11445.807414] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11450.052040] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11465.719296] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11465.726378] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11465.742331] Lustre: Skipped 3 previous similar messages [11465.763930] Lustre: Skipped 2 previous similar messages [11470.821917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11470.834205] Lustre: Skipped 2 previous similar messages [11471.347657] Lustre: server umount lustre-MDT0000 complete [11475.942827] LustreError: 363936:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11475.972158] LustreError: 363936:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [11476.014516] LustreError: 362177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787877033 with bad export cookie 10864596379666665483 [11476.033230] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11476.041161] LustreError: 362177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11476.527659] Lustre: server umount lustre-MDT0001 complete [11492.276974] Lustre: server umount lustre-OST0000 complete [11517.375758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11517.732867] LustreError: 366114:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11517.781384] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11523.197555] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [11537.238921] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11546.250849] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [11558.925710] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11567.305755] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11574.145046] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11574.632911] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11574.640609] Lustre: Skipped 1 previous similar message [11581.591108] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [11588.026965] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11589.982782] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11597.285994] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11597.292185] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11597.301673] Lustre: Skipped 1 previous similar message [11602.424650] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11602.435960] Lustre: Skipped 2 previous similar messages [11602.571699] Lustre: server umount lustre-OST0000 complete [11607.521881] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11607.531226] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11607.535928] Lustre: Skipped 1 previous similar message [11613.850640] Lustre: server umount lustre-MDT0000 complete [11617.763545] LustreError: 366109:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11617.790416] LustreError: 366109:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [11620.076910] LustreError: 366095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787877177 with bad export cookie 10864596379666666505 [11620.091954] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11620.415835] Lustre: server umount lustre-MDT0001 complete [11639.577882] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [11643.123962] Key type lgssc unregistered [11643.547778] LNet: 369671:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11643.562319] LNetError: 369671:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11643.609797] LNet: Removed LNI 192.168.204.126@tcp [11644.647787] Key type .llcrypt unregistered [11644.653812] Key type ._llcrypt unregistered [11672.848738] Key type ._llcrypt registered [11672.852091] Key type .llcrypt registered [11673.070783] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [11674.489972] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11674.543091] alg: No test for adler32 (adler32-zlib) [11676.114512] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [11676.552358] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [11678.296765] Key type lgssc registered [11680.134389] Lustre: Echo OBD driver; http://www.lustre.org/ [11690.840605] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11690.849500] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11690.856523] LustreError: 371117:0:(osd_handler.c:8517:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11690.870848] LustreError: 371117:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11690.875980] LustreError: 371117:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11690.880646] LustreError: 371117:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11690.884213] LustreError: 371117:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11698.673525] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11698.678657] LustreError: 371514:0:(osd_handler.c:8517:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11698.696521] LustreError: 371514:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11698.703299] LustreError: 371514:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11698.710954] LustreError: 371514:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11698.721378] LustreError: 371514:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11718.163595] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [11721.341597] Key type lgssc unregistered [11721.620606] LNet: 372522:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11721.632624] LNetError: 372522:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11721.659253] LNet: Removed LNI 192.168.204.126@tcp [11722.395267] Key type .llcrypt unregistered [11722.398862] Key type ._llcrypt unregistered [11742.260945] Key type ._llcrypt registered [11742.264497] Key type .llcrypt registered [11742.371342] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [11758.894565] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [11760.108069] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11760.132319] alg: No test for adler32 (adler32-zlib) [11761.368838] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [11761.669854] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [11763.328196] Key type lgssc registered [11764.520428] Lustre: Echo OBD driver; http://www.lustre.org/ [11804.459665] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11804.486976] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11805.824792] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11805.852472] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11805.909638] Lustre: lustre-MDT0000: new disk, initializing [11805.985772] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11806.002239] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11810.492657] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [11820.836090] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11820.968629] Lustre: 375680:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11821.014732] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11821.018781] Lustre: Skipped 1 previous similar message [11821.087984] Lustre: lustre-MDT0001: new disk, initializing [11821.156844] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11821.186650] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11821.198722] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11825.135644] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [11837.314714] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11848.307487] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11860.137804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11860.763568] Lustre: lustre-OST0000: new disk, initializing [11860.767739] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11860.780854] Lustre: 377650:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11860.890836] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11870.271964] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11870.282868] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11870.291450] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [11870.508256] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11882.495942] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11888.261553] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11888.430272] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11893.702696] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11894.019272] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11901.413715] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11901.440711] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11901.450591] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11901.464561] Lustre: Skipped 1 previous similar message [11901.483638] Lustre: Skipped 1 previous similar message [11903.708082] Lustre: server umount lustre-OST0000 complete [11908.068593] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11908.088227] LustreError: Skipped 1 previous similar message [11908.102512] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11908.139133] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11911.653811] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11911.657217] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11913.922742] Lustre: server umount lustre-MDT0000 complete [11917.795081] LustreError: 375671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787877474 with bad export cookie 16764094675415901788 [11917.802448] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11917.821557] LustreError: 375671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11918.104414] Lustre: server umount lustre-MDT0001 complete [11927.559155] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11929.772928] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 20:38:04 (1787877484) [11937.516978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11938.023806] LustreError: 380113:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11938.038325] LustreError: 380113:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [11938.109343] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11942.154476] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [11943.401407] LustreError: 380114:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11948.518631] LustreError: 380113:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11953.445662] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11953.633336] LustreError: 380114:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11953.938290] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11959.698814] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [11970.000702] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11976.370753] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11983.164958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11983.521582] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11989.050651] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [11996.942088] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12003.140513] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12003.293526] Lustre: lustre-OST0001: new disk, initializing [12003.296575] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [12003.301988] Lustre: 382953:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12003.381376] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12011.194190] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12011.216473] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12011.400627] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12012.021142] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [12023.799432] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12030.962440] Lustre: lustre-OST0001: Not available for connect from 192.168.204.26@tcp (stopping) [12030.992792] Lustre: Skipped 1 previous similar message [12031.973732] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12031.990309] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12033.857772] Lustre: server umount lustre-OST0001 complete [12036.023512] LustreError: 381934:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12047.341266] LustreError: 381937:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12047.369914] LustreError: 381937:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [12052.460400] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12052.461087] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12052.467745] Lustre: Skipped 2 previous similar messages [12052.490185] Lustre: Skipped 3 previous similar messages [12062.704325] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12062.712305] Lustre: Skipped 3 previous similar messages [12066.272156] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12066.495062] Lustre: server umount lustre-OST0000 complete [12071.905517] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12071.917077] LustreError: Skipped 1 previous similar message [12071.921249] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12077.073858] Lustre: server umount lustre-MDT0000 complete [12078.050380] LustreError: 382556:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12078.070300] LustreError: 382556:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 7 previous similar messages [12080.733562] LustreError: 381958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787877637 with bad export cookie 16764094675415902677 [12080.738736] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12080.751958] LustreError: 381958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12087.285640] Lustre: server umount lustre-MDT0001 complete [12100.769367] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [12103.540784] Key type lgssc unregistered [12103.992144] LNet: 384945:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12104.008926] LNetError: 384945:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12104.035826] LNet: Removed LNI 192.168.204.126@tcp [12105.092931] Key type .llcrypt unregistered [12105.094533] Key type ._llcrypt unregistered [12162.736775] Key type ._llcrypt registered [12162.740784] Key type .llcrypt registered [12162.868270] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [12164.078874] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12164.269078] alg: No test for adler32 (adler32-zlib) [12165.255270] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [12165.511623] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [12167.120289] Key type lgssc registered [12168.335904] Lustre: Echo OBD driver; http://www.lustre.org/ [12177.623707] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12177.638038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12179.013957] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12179.058138] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12179.338266] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12185.248774] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [12195.903714] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12196.093701] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12196.142472] Lustre: 387747:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12196.695828] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12202.483187] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [12212.635921] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12221.559710] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12227.565154] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12227.583787] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12227.601424] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12228.578602] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12228.582498] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12230.785673] Lustre: server umount lustre-MDT0000 complete [12235.167822] LustreError: 387740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787877792 with bad export cookie 12601969608348957682 [12235.179090] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12235.186711] LustreError: 387740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12235.685380] Lustre: server umount lustre-MDT0001 complete [12247.081540] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 20:43:21 (1787877801) [12254.313513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12254.867265] LustreError: 390260:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12254.881881] LustreError: 390260:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [12254.928114] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12259.908225] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [12260.325481] LustreError: 390261:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12265.443175] LustreError: 390260:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12269.401047] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12269.723249] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12274.475392] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [12284.453560] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12291.526679] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12299.509460] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12299.706851] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [12299.995832] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12306.673722] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [12314.183563] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12324.139312] Lustre: Permanently deactivating lustre-OST0000 [12324.172785] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [12333.122023] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12336.243561] LustreError: 393247:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [12336.278858] LustreError: 393247:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12336.306049] LustreError: 393247:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12340.299773] LustreError: 393296:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12340.316148] LustreError: 393296:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12348.846224] Lustre: server umount lustre-OST0000 complete [12352.930490] LustreError: 393547:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12352.935884] LustreError: 393547:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12357.091482] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12357.105779] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12357.141632] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12357.157426] Lustre: Skipped 1 previous similar message [12359.140861] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12359.148657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12359.191632] Lustre: Skipped 1 previous similar message [12364.261548] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12364.272531] Lustre: Skipped 1 previous similar message [12366.418530] Lustre: server umount lustre-MDT0000 complete [12369.383521] LustreError: 390260:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12369.399796] LustreError: 390260:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12370.552484] LustreError: 390241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787877927 with bad export cookie 12601969608348958263 [12370.554464] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12370.570696] LustreError: 390241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12380.467444] Lustre: server umount lustre-MDT0001 complete [12389.902638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12394.874378] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [12397.712672] LustreError: 394707:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [12404.729842] Lustre: 395046:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.126@tcp [12407.327729] Lustre: server umount MGS complete [12419.705207] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12420.283301] LustreError: 395770:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12420.304057] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12420.308882] Lustre: Skipped 1 previous similar message [12420.364622] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12425.174604] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [12430.835819] LustreError: 395771:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12430.861351] LustreError: 395771:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12436.320718] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12436.720739] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12436.826985] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12444.319615] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [12458.707542] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12466.422884] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12468.834859] Lustre: Permanently reactivating lustre-OST0000 [12468.848127] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12468.856114] Lustre: Skipped 2 previous similar messages [12496.864744] LustreError: 397532:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12506.480086] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12507.105894] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12514.188740] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [12524.756491] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12531.686119] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12531.692489] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12531.696079] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12531.701521] LustreError: Skipped 1 previous similar message [12531.712152] Lustre: Skipped 1 previous similar message [12537.871283] Lustre: server umount lustre-OST0000 complete [12544.482742] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12544.494620] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12544.515099] Lustre: Skipped 1 previous similar message [12548.909932] Lustre: server umount lustre-MDT0000 complete [12552.173025] LustreError: 395766:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12552.203940] LustreError: 395766:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [12553.960762] LustreError: 396506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787878110 with bad export cookie 12601969608348959313 [12553.974574] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12553.986665] LustreError: 396506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12554.026454] LustreError: 387069:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting [12554.597858] Lustre: server umount lustre-MDT0001 complete [12570.096303] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [12574.060358] Key type lgssc unregistered [12574.463141] LNet: 400096:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12574.480758] LNetError: 400096:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12574.495915] LNet: Removed LNI 192.168.204.126@tcp [12575.620931] Key type .llcrypt unregistered [12575.622975] Key type ._llcrypt unregistered [12605.615849] Key type ._llcrypt registered [12605.618593] Key type .llcrypt registered [12605.733763] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [12623.755790] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [12625.472806] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12625.602753] alg: No test for adler32 (adler32-zlib) [12627.500484] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [12627.970589] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [12629.729436] Key type lgssc registered [12632.075350] Lustre: Echo OBD driver; http://www.lustre.org/ [12674.902181] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12674.975761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12676.460459] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12676.522133] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12676.689347] Lustre: lustre-MDT0000: new disk, initializing [12676.756724] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12676.779793] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12682.863535] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [12698.974193] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12699.082419] Lustre: 403260:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12699.140130] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12699.143225] Lustre: Skipped 1 previous similar message [12699.213795] Lustre: lustre-MDT0001: new disk, initializing [12699.373874] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12699.469905] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12699.499172] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12706.681325] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [12718.791440] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12726.279427] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12735.108360] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12735.458879] Lustre: lustre-OST0000: new disk, initializing [12735.467819] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12735.477128] Lustre: 405233:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12735.595861] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12741.754855] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12741.771688] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12741.887536] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12743.357708] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [12754.844704] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12760.741912] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12761.035356] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12766.467535] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12766.749697] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12772.835411] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12772.846967] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12772.865681] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12772.880409] Lustre: Skipped 1 previous similar message [12775.138611] Lustre: server umount lustre-OST0000 complete [12781.537859] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12781.546927] LustreError: Skipped 1 previous similar message [12781.558038] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12781.578146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12781.584672] Lustre: Skipped 1 previous similar message [12783.074241] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12783.075369] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12783.104846] Lustre: Skipped 1 previous similar message [12788.194268] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12788.207579] Lustre: Skipped 1 previous similar message [12792.291235] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12792.302701] Lustre: Skipped 1 previous similar message [12794.337204] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12794.505257] Lustre: server umount lustre-MDT0000 complete [12798.560588] LustreError: 404058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787878355 with bad export cookie 1350931334926478068 [12798.567394] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12798.570820] LustreError: 404058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12799.076628] Lustre: server umount lustre-MDT0001 complete [12810.661900] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12812.764907] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 20:52:47 (1787878367) [12822.029318] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12822.525979] LustreError: 407695:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12822.548037] LustreError: 407695:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [12822.587495] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12827.618433] LustreError: 407696:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12827.984493] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [12832.737351] LustreError: 407695:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12836.155041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12836.561622] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12840.196618] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [12847.629463] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12853.825256] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12860.684829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12860.941991] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12867.632475] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [12875.846747] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12877.748235] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12880.459982] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12887.522088] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12887.523582] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12887.536366] LustreError: Skipped 1 previous similar message [12887.538406] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12887.584736] Lustre: Skipped 1 previous similar message [12892.338827] Lustre: server umount lustre-OST0000 complete [12897.764510] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12897.769753] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12903.510888] Lustre: server umount lustre-MDT0000 complete [12908.003767] LustreError: 407695:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12908.030980] LustreError: 407695:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12908.205444] LustreError: 407675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787878465 with bad export cookie 1350931334926478950 [12908.210159] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12908.212146] LustreError: 407675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12908.487509] Lustre: server umount lustre-MDT0001 complete [12919.959087] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [12923.294708] Key type lgssc unregistered [12923.593804] LNet: 411425:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12923.602221] LNetError: 411425:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12923.628745] LNet: Removed LNI 192.168.204.126@tcp [12924.739845] Key type .llcrypt unregistered [12924.746838] Key type ._llcrypt unregistered [12957.230274] Key type ._llcrypt registered [12957.233935] Key type .llcrypt registered [12957.462459] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [12958.117542] Key type .llcrypt unregistered [12958.121318] Key type ._llcrypt unregistered [12977.925346] Key type ._llcrypt registered [12977.927831] Key type .llcrypt registered [12978.038949] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [12979.380414] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12979.498846] alg: No test for adler32 (adler32-zlib) [12980.604129] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [12980.900216] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [12982.656989] Key type lgssc registered [12984.344815] Lustre: Echo OBD driver; http://www.lustre.org/ [12993.933859] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12994.006342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12995.663340] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13002.025677] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [13005.814461] LustreError: 413652:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13005.838226] LustreError: 413652:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [13010.918905] LustreError: 413653:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13014.152505] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13014.618538] LustreError: 413652:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13014.650928] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13014.651968] LustreError: 413652:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [13020.130048] LustreError: 413653:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13020.144934] LustreError: 413653:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [13021.173135] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [13021.760784] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [13026.276256] LustreError: 413653:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13026.287856] LustreError: 413653:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [13030.702452] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13036.518877] LustreError: 413653:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13036.536195] LustreError: 413653:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [13039.831396] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13040.475685] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13046.605423] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [13057.000330] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13057.014045] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13057.041755] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13061.092981] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13061.115100] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13061.116370] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13061.136318] Lustre: Skipped 1 previous similar message [13066.216984] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13066.240137] Lustre: Skipped 1 previous similar message [13071.332897] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13071.339437] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13071.363575] Lustre: Skipped 1 previous similar message [13071.645803] Lustre: server umount lustre-OST0000 complete [13076.450248] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13076.455214] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13076.470520] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13076.476195] Lustre: Skipped 1 previous similar message [13076.488350] Lustre: Skipped 1 previous similar message [13081.971545] Lustre: server umount lustre-MDT0000 complete [13086.690861] LustreError: 415391:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13086.718244] LustreError: 415391:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [13087.230580] LustreError: 415398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787878644 with bad export cookie 2627478905765856462 [13087.232044] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13087.245437] LustreError: 415398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13087.574309] Lustre: server umount lustre-MDT0001 complete [13100.477913] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [13105.035609] Key type lgssc unregistered [13105.690826] LNet: 416809:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13105.700873] LNetError: 416809:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13105.743888] LNet: Removed LNI 192.168.204.126@tcp [13107.249385] Key type .llcrypt unregistered [13107.251837] Key type ._llcrypt unregistered [13133.446398] Key type ._llcrypt registered [13133.448653] Key type .llcrypt registered [13133.532361] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [13134.220059] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13134.315051] alg: No test for adler32 (adler32-zlib) [13135.357497] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [13135.605491] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [13137.392499] Key type lgssc registered [13138.739335] Lustre: Echo OBD driver; http://www.lustre.org/ [13148.247954] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13148.271954] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13163.873074] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [13168.992329] LustreError: 418202:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.126@tcp: failed processing log, type 4: rc = -110 [13195.680554] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13200.823124] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [13208.170426] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13216.578898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13217.270951] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13217.368167] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [13222.375396] LustreError: 418228:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13222.407709] LustreError: 418228:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [13223.018787] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [13227.491022] LustreError: 418227:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13230.592100] LustreError: 418228:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13230.627436] LustreError: 418228:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [13234.355276] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13234.926691] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13241.675480] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [13251.344328] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13257.637813] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13265.893908] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13265.918077] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13270.498259] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13270.503984] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13270.514536] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13276.133687] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13276.146966] Lustre: Skipped 2 previous similar messages [13280.224362] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13280.366765] Lustre: server umount lustre-OST0000 complete [13285.864752] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13285.872578] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13285.894553] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13285.901097] Lustre: Skipped 1 previous similar message [13290.865085] Lustre: server umount lustre-MDT0000 complete [13291.501215] LustreError: 419242:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13291.534420] LustreError: 419242:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [13295.015789] LustreError: 418210:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787878852 with bad export cookie 10846151393770188576 [13295.024542] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13295.029028] LustreError: 418210:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13295.266878] Lustre: server umount lustre-MDT0001 complete [13306.967137] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [13310.743270] Key type lgssc unregistered [13311.092838] LNet: 421788:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13311.102693] LNetError: 421788:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13311.126203] LNet: Removed LNI 192.168.204.126@tcp [13312.496195] Key type .llcrypt unregistered [13312.498070] Key type ._llcrypt unregistered [13339.360208] Key type ._llcrypt registered [13339.363168] Key type .llcrypt registered [13339.453750] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [13340.335096] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13340.352054] alg: No test for adler32 (adler32-zlib) [13341.479682] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [13341.760058] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [13343.408248] Key type lgssc registered [13344.614467] Lustre: Echo OBD driver; http://www.lustre.org/ [13356.237241] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13356.278809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13358.227958] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13363.682822] LustreError: 423208:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13363.709330] LustreError: 423208:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [13364.458255] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [13368.810909] LustreError: 423209:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13373.634197] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13373.926114] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13379.406739] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [13386.843491] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13392.910909] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13398.611860] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13398.962532] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13403.052918] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [13404.213939] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [13411.702841] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13418.147075] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13418.157078] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13418.164203] Lustre: Skipped 2 previous similar messages [13418.181540] Lustre: Skipped 2 previous similar messages [13421.294850] Lustre: server umount lustre-MDT0000 complete [13423.542857] LustreError: 424514:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13424.150673] LustreError: 423189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.26@tcp arrived at 1787878981 with bad export cookie 14244915235676793038 [13424.169452] LustreError: 423189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13426.588411] LustreError: 423188:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787878983 with bad export cookie 14244915235676792149 [13426.594916] LustreError: 423188:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13426.596630] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13428.723711] LustreError: 423209:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13428.733473] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13428.766055] LustreError: 423209:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [13428.816658] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13428.824186] Lustre: Skipped 1 previous similar message [13431.778566] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13433.429216] Lustre: server umount lustre-OST0000 complete [13436.915842] LustreError: 423204:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13436.946033] LustreError: 423204:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [13444.832455] LustreError: 423939:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000001c9fcfb7 x1874726903380992/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13445.267772] Lustre: server umount lustre-MDT0001 complete [13457.324711] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [13460.403992] Key type lgssc unregistered [13460.739640] LNet: 426897:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13460.743890] LNetError: 426897:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13460.766541] LNet: Removed LNI 192.168.204.126@tcp [13461.527182] Key type .llcrypt unregistered [13461.529366] Key type ._llcrypt unregistered [13486.173919] Key type ._llcrypt registered [13486.188761] Key type .llcrypt registered [13486.323321] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [13487.765410] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13487.780180] alg: No test for adler32 (adler32-zlib) [13488.955491] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [13489.271679] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [13491.000141] Key type lgssc registered [13492.219516] Lustre: Echo OBD driver; http://www.lustre.org/ [13500.502304] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13500.528180] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13502.078290] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13506.714358] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [13512.680143] LustreError: 428313:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13512.705741] LustreError: 428313:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [13514.722617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13515.054930] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13519.364634] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [13526.827566] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13533.617573] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13541.694347] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13542.119248] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13547.260059] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [13547.926813] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [13556.535554] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13561.314590] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13561.326197] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13561.338289] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13564.339409] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.26@tcp (stopping) [13564.906225] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13567.499725] Lustre: server umount lustre-MDT0001 complete [13569.472405] LustreError: 428307:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13575.138224] LustreError: 428313:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13575.140427] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13575.168389] LustreError: 428313:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [13575.194922] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13575.201036] Lustre: Skipped 2 previous similar messages [13580.260285] LustreError: 429040:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13580.262163] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13580.640567] Lustre: server umount lustre-OST0000 complete [13585.286952] Lustre: server umount lustre-MDT0000 complete [13600.289495] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [13603.735992] Key type lgssc unregistered [13604.152812] LNet: 432001:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13604.157449] LNetError: 432001:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13604.170223] LNet: Removed LNI 192.168.204.126@tcp [13605.106775] Key type .llcrypt unregistered [13605.109364] Key type ._llcrypt unregistered [13630.969965] Key type ._llcrypt registered [13630.971719] Key type .llcrypt registered [13631.052178] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [13647.293581] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [13649.268609] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13649.368695] alg: No test for adler32 (adler32-zlib) [13650.670762] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [13651.170611] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [13652.928420] Key type lgssc registered [13654.238724] Lustre: Echo OBD driver; http://www.lustre.org/ [13695.591808] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [13715.099566] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13715.196979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13716.930928] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13717.018445] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13717.299523] Lustre: lustre-MDT0000: new disk, initializing [13717.484322] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13717.510228] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13723.612337] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [13743.853885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13744.110727] Lustre: lustre-OST0000: new disk, initializing [13744.115481] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13744.118224] Lustre: Skipped 1 previous similar message [13744.121617] Lustre: 437045:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13744.183146] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13750.924458] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [13751.402415] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13751.407953] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13751.437938] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13761.558782] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13777.871964] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13778.104057] Lustre: 436027:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13778.217297] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13778.269479] Lustre: lustre-MDT0001: new disk, initializing [13778.455253] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13778.552740] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13778.577356] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13783.556581] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [13793.594566] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13793.833699] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13797.934465] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13798.156378] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13803.481909] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.26@tcp (stopping) [13803.493365] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13803.497772] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13803.538498] Lustre: Skipped 1 previous similar message [13808.568892] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.26@tcp (stopping) [13808.582875] Lustre: Skipped 2 previous similar messages [13809.460066] Lustre: server umount lustre-MDT0001 complete [13818.855510] LustreError: 436038:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13818.883678] LustreError: 436038:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [13823.975961] LustreError: 436037:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13824.024289] LustreError: 436037:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [13829.094585] LustreError: 438704:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13829.136264] LustreError: 438704:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [13833.187299] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13833.202531] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13833.238381] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13833.244746] Lustre: Skipped 2 previous similar messages [13834.212375] LustreError: 436038:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13836.611206] Lustre: server umount lustre-OST0000 complete [13839.329432] LustreError: 436037:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13847.933938] Lustre: server umount lustre-MDT0000 complete [13868.311883] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [13872.812755] Key type lgssc unregistered [13873.130641] LNet: 440810:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13873.139823] LNetError: 440810:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13873.175661] LNet: Removed LNI 192.168.204.126@tcp [13874.398520] Key type .llcrypt unregistered [13874.403515] Key type ._llcrypt unregistered [13924.290751] Key type ._llcrypt registered [13924.295430] Key type .llcrypt registered [13924.510648] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [13924.803432] Key type .llcrypt unregistered [13924.804773] Key type ._llcrypt unregistered [13942.807314] Key type ._llcrypt registered [13942.812298] Key type .llcrypt registered [13942.990720] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [13944.090385] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13944.174049] alg: No test for adler32 (adler32-zlib) [13945.196167] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [13945.427984] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [13947.169077] Key type lgssc registered [13948.757013] Lustre: Echo OBD driver; http://www.lustre.org/ [13981.443380] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13981.467647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13982.961366] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13983.036281] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13983.255739] Lustre: lustre-MDT0000: new disk, initializing [13983.410852] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13983.451390] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13989.843422] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [14001.540540] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14001.743217] Lustre: 444193:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14001.846834] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14001.853910] Lustre: Skipped 1 previous similar message [14001.959767] Lustre: lustre-MDT0001: new disk, initializing [14002.039096] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14002.081159] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14002.105253] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14006.585958] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [14016.912206] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14023.137707] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14030.585920] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14030.877161] Lustre: lustre-OST0000: new disk, initializing [14030.880451] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14030.886976] Lustre: 446163:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14030.957362] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14036.589920] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14036.607034] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14036.791422] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14038.678525] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [14051.796683] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14062.562074] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14062.578049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14062.589458] Lustre: Skipped 1 previous similar message [14064.985320] Lustre: server umount lustre-MDT0000 complete [14068.830826] LustreError: 444184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787879625 with bad export cookie 14261292385674837904 [14068.839454] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14068.853919] LustreError: 444184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14069.239245] Lustre: server umount lustre-MDT0001 complete [14082.434604] Lustre: server umount lustre-OST0000 complete [14093.161626] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 21:14:08 (1787879648) [14101.652417] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14102.021116] LustreError: 448188:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14102.053264] LustreError: 448188:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [14102.132646] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14107.013701] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [14107.617462] LustreError: 448189:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14112.751434] LustreError: 448188:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14115.925071] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14116.285473] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14120.057217] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [14129.202608] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14137.855587] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14159.812155] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14160.338450] Lustre: Found index 0 for lustre-OST0000, updating log [14160.634806] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14167.978392] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [14175.031487] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [14178.242570] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14185.457862] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14185.461610] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14185.465128] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14185.512588] Lustre: Skipped 3 previous similar messages [14186.594080] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14186.616779] Lustre: Skipped 1 previous similar message [14190.241720] Lustre: server umount lustre-OST0000 complete [14208.481870] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14208.482775] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14208.489927] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14208.514923] Lustre: Skipped 1 previous similar message [14212.781423] Lustre: server umount lustre-MDT0000 complete [14213.617911] LustreError: 448916:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14213.631545] LustreError: 448916:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [14218.037812] LustreError: 448169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787879775 with bad export cookie 14261292385674839290 [14218.042349] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14218.045401] LustreError: 448169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14218.422411] Lustre: server umount lustre-MDT0001 complete [14221.985585] LNet: 1 local NIs in recovery (showing 1): 192.168.204.126@tcp [14229.886541] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 21:16:24 (1787879784) [14850.962361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14851.344459] LustreError: 454599:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14851.371736] LustreError: 454599:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [14851.419515] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14855.907683] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [14856.674798] LustreError: 454600:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14861.795057] LustreError: 454599:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14864.014630] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14864.429558] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14868.794607] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [14876.181407] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14881.853100] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14902.572281] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14902.872204] Lustre: Found index 0 for lustre-OST0000, updating log [14903.204103] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14909.451209] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [14912.160201] LNet: 1 local NIs in recovery (showing 1): 192.168.204.126@tcp [14917.970751] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14922.728299] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14922.742770] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14922.770922] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14922.780564] Lustre: Skipped 1 previous similar message [14926.667097] Lustre: server umount lustre-OST0000 complete [14930.916227] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14930.931934] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14930.947898] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14932.004441] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14932.009633] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14934.124454] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14934.133108] Lustre: Skipped 4 previous similar messages [14938.213150] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14938.223378] Lustre: Skipped 6 previous similar messages [14945.248129] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14945.391837] Lustre: server umount lustre-MDT0000 complete [14945.450254] LustreError: 454599:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14945.460278] LustreError: 454599:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [14949.538328] LustreError: 454600:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14949.547666] LustreError: 454600:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 7 previous similar messages [14949.550728] LustreError: 454581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787880506 with bad export cookie 14261292385674840382 [14949.555425] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14949.566512] LustreError: 454581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14950.027183] Lustre: server umount lustre-MDT0001 complete [14981.991303] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [14992.669666] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [15034.502627] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 21:29:49 (1787880589) [15057.985161] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [15061.377365] Key type lgssc unregistered [15061.806192] LNet: 461223:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15061.826218] LNetError: 461223:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15165.291256] LNet: Removed LNI 192.168.204.126@tcp [15166.016591] Key type .llcrypt unregistered [15166.018144] Key type ._llcrypt unregistered [15182.488886] Key type ._llcrypt registered [15182.492791] Key type .llcrypt registered [15182.566167] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [15183.246533] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15183.278090] alg: No test for adler32 (adler32-zlib) [15184.424241] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [15184.758286] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [15186.409974] Key type lgssc registered [15187.295497] Lustre: Echo OBD driver; http://www.lustre.org/ [15191.518568] LNet: Added LNI 192.168.204.126@tcp100 [8/256/0/180] [15192.530761] LNet: Added LNI 192.168.204.126@tcp101 [8/256/0/180] [15193.358743] LNet: Added LNI 192.168.204.126@tcp102 [8/256/0/180] [15194.470934] LNet: Added LNI 192.168.204.126@tcp103 [8/256/0/180] [15195.442153] LNet: Added LNI 192.168.204.126@tcp104 [8/256/0/180] [15196.529396] LNet: Added LNI 192.168.204.126@tcp105 [8/256/0/180] [15197.697047] LNet: Added LNI 192.168.204.126@tcp106 [8/256/0/180] [15198.710273] LNet: Added LNI 192.168.204.126@tcp107 [8/256/0/180] [15200.022468] LNet: Added LNI 192.168.204.126@tcp108 [8/256/0/180] [15201.420632] LNet: Added LNI 192.168.204.126@tcp109 [8/256/0/180] [15202.960956] LNet: Added LNI 192.168.204.126@tcp110 [8/256/0/180] [15203.956564] LNet: Added LNI 192.168.204.126@tcp111 [8/256/0/180] [15205.201076] LNet: Added LNI 192.168.204.126@tcp112 [8/256/0/180] [15206.379600] LNet: Added LNI 192.168.204.126@tcp113 [8/256/0/180] [15207.394493] LNet: Added LNI 192.168.204.126@tcp114 [8/256/0/180] [15208.489896] LNet: Added LNI 192.168.204.126@tcp115 [8/256/0/180] [15209.577475] LNet: Added LNI 192.168.204.126@tcp116 [8/256/0/180] [15210.633257] LNet: Added LNI 192.168.204.126@tcp117 [8/256/0/180] [15211.589523] LNet: Added LNI 192.168.204.126@tcp118 [8/256/0/180] [15212.787534] LNet: Added LNI 192.168.204.126@tcp119 [8/256/0/180] [15213.931153] LNet: Added LNI 192.168.204.126@tcp120 [8/256/0/180] [15215.107975] LNet: Added LNI 192.168.204.126@tcp121 [8/256/0/180] [15216.267556] LNet: Added LNI 192.168.204.126@tcp122 [8/256/0/180] [15217.351941] LNet: Added LNI 192.168.204.126@tcp123 [8/256/0/180] [15218.466273] LNet: Added LNI 192.168.204.126@tcp124 [8/256/0/180] [15219.499063] LNet: Added LNI 192.168.204.126@tcp125 [8/256/0/180] [15220.588349] LNet: Added LNI 192.168.204.126@tcp126 [8/256/0/180] [15221.557311] LNet: Added LNI 192.168.204.126@tcp127 [8/256/0/180] [15222.734941] LNet: Added LNI 192.168.204.126@tcp128 [8/256/0/180] [15223.961534] LNet: Added LNI 192.168.204.126@tcp129 [8/256/0/180] [15225.081878] LNet: Added LNI 192.168.204.126@tcp130 [8/256/0/180] [15226.483268] LNet: Added LNI 192.168.204.126@tcp131 [8/256/0/180] [15227.567087] LNet: Added LNI 192.168.204.126@tcp132 [8/256/0/180] [15228.658148] LNet: Added LNI 192.168.204.126@tcp133 [8/256/0/180] [15229.572276] LNet: Added LNI 192.168.204.126@tcp134 [8/256/0/180] [15230.469641] LNet: Added LNI 192.168.204.126@tcp135 [8/256/0/180] [15238.059064] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15238.087860] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15239.177239] Lustre: 464253:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.126@tcp: can't add NID 192.168.204.126@tcp130: rc = -75 [15239.257111] LNetError: 461712:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.126 [15239.271763] LNetError: Protocol error connecting to 192.168.204.126@tcp119 at host 192.168.204.126:988: is it running a compatible version of Lustre? [15239.289400] LNetError: Skipped 8 previous similar messages [15239.385514] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15239.408615] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15239.457740] Lustre: lustre-MDT0000: new disk, initializing [15239.537682] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15239.577429] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15240.096743] LNet: 6 local NIs in recovery (showing 6): 192.168.204.126@tcp125, 192.168.204.126@tcp127, 192.168.204.126@tcp129, 192.168.204.126@tcp128, 192.168.204.126@tcp126, 192.168.204.126@tcp119 [15240.100290] LNetError: 464276:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.126 [15240.132786] LNetError: 464276:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 8 previous similar messages [15240.141471] LNetError: Protocol error connecting to 192.168.204.126@tcp115 at host 192.168.204.126:988: is it running a compatible version of Lustre? [15240.152278] LNetError: Skipped 3 previous similar messages [15244.901615] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [15256.612220] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15256.753907] Lustre: 464268:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15256.805722] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15256.815390] Lustre: Skipped 1 previous similar message [15256.909443] Lustre: lustre-MDT0001: new disk, initializing [15256.945082] Lustre: 465072:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.126@tcp: can't add NID 192.168.204.126@tcp130: rc = -75 [15256.949250] Lustre: 465072:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15257.031292] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15257.084958] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15257.101670] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15262.551748] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [15264.811610] Lustre: 465434:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.126@tcp: can't add NID 192.168.204.126@tcp130: rc = -75 [15264.818409] Lustre: 465434:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15269.733046] LNetError: 461709:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.126 [15269.745952] LNetError: 461709:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 3 previous similar messages [15269.763743] LNetError: Protocol error connecting to 192.168.204.126@tcp115 at host 192.168.204.126:988: is it running a compatible version of Lustre? [15272.805822] LNetError: 461711:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.126 [15272.821286] LNetError: 461711:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 2 previous similar messages [15272.840131] LNetError: Protocol error connecting to 192.168.204.126@tcp126 at host 192.168.204.126:988: is it running a compatible version of Lustre? [15272.878261] LNetError: Skipped 2 previous similar messages [15273.550420] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15276.904072] LNetError: 464279:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.126 [15276.920718] LNetError: 464279:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 1 previous similar message [15276.929860] LNetError: Protocol error connecting to 192.168.204.126@tcp119 at host 192.168.204.126:988: is it running a compatible version of Lustre? [15276.937963] LNetError: Skipped 1 previous similar message [15280.808855] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15285.095125] LNetError: 461710:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.126 [15285.107797] LNetError: 461710:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 4 previous similar messages [15285.116309] LNetError: Protocol error connecting to 192.168.204.126@tcp126 at host 192.168.204.126:988: is it running a compatible version of Lustre? [15285.133327] LNetError: Skipped 4 previous similar messages [15288.596158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15288.943673] Lustre: lustre-OST0000: new disk, initializing [15288.947734] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15288.955120] Lustre: 466244:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15289.046836] Lustre: 466249:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.126@tcp: can't add NID 192.168.204.126@tcp130: rc = -75 [15289.055635] Lustre: 466249:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15289.122889] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15296.748992] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [15298.623486] Lustre: 466687:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.126@tcp: can't add NID 192.168.204.126@tcp130: rc = -75 [15298.631280] Lustre: 466687:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15298.697984] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15298.710343] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15298.883469] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15301.475155] LNetError: 461709:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.126 [15301.489390] LNetError: 461709:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 7 previous similar messages [15301.499609] LNetError: Protocol error connecting to 192.168.204.126@tcp124 at host 192.168.204.126:988: is it running a compatible version of Lustre? [15301.512066] LNetError: Skipped 7 previous similar messages [15307.473161] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15319.521733] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15319.525864] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15319.528566] LustreError: Skipped 1 previous similar message [15319.531316] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15319.566170] Lustre: Skipped 1 previous similar message [15321.684166] Lustre: server umount lustre-OST0000 complete [15328.744752] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15328.759082] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15328.768382] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15328.770987] Lustre: Skipped 1 previous similar message [15329.767850] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15329.781783] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15329.790347] Lustre: Skipped 1 previous similar message [15331.732442] Lustre: server umount lustre-MDT0000 complete [15334.241363] LNetError: 461710:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.126 [15334.248553] LNetError: 461710:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 9 previous similar messages [15334.259411] LNetError: Protocol error connecting to 192.168.204.126@tcp123 at host 192.168.204.126:988: is it running a compatible version of Lustre? [15334.270822] LNetError: Skipped 9 previous similar messages [15335.772981] LustreError: 464263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787880892 with bad export cookie 4489564727223228426 [15335.777658] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15335.783161] LustreError: 464263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15336.191868] Lustre: server umount lustre-MDT0001 complete [15347.877703] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [15350.333575] Key type lgssc unregistered [15350.649797] LNet: 468163:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15350.661547] LNetError: 468163:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15350.668562] LNet: Removed LNI 192.168.204.126@tcp [15351.712599] LNet: Removed LNI 192.168.204.126@tcp100 [15351.718623] LNetError: 468163:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15351.724058] LNetError: 468163:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15351.732066] LNet: Removed LNI 192.168.204.126@tcp101 [15352.800729] LNet: Removed LNI 192.168.204.126@tcp102 [15352.804120] LNetError: 468163:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15352.812296] LNetError: 468163:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15352.834686] LNet: Removed LNI 192.168.204.126@tcp103 [15353.888338] LNet: Removed LNI 192.168.204.126@tcp104 [15354.912289] LNet: Removed LNI 192.168.204.126@tcp105 [15354.918631] LNetError: 468163:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15354.931418] LNetError: 468163:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [15354.942524] LNet: Removed LNI 192.168.204.126@tcp106 [15356.000457] LNet: Removed LNI 192.168.204.126@tcp107 [15357.024402] LNet: Removed LNI 192.168.204.126@tcp108 [15358.048790] LNet: Removed LNI 192.168.204.126@tcp109 [15359.073084] LNet: Removed LNI 192.168.204.126@tcp110 [15359.077178] LNetError: 468163:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15359.093290] LNetError: 468163:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [15359.103309] LNet: Removed LNI 192.168.204.126@tcp111 [15360.160676] LNet: Removed LNI 192.168.204.126@tcp112 [15361.184207] LNet: Removed LNI 192.168.204.126@tcp113 [15362.208173] LNet: Removed LNI 192.168.204.126@tcp114 [15363.232107] LNet: Removed LNI 192.168.204.126@tcp115 [15364.256449] LNet: Removed LNI 192.168.204.126@tcp116 [15365.284049] LNet: Removed LNI 192.168.204.126@tcp117 [15366.304541] LNet: Removed LNI 192.168.204.126@tcp118 [15367.328341] LNet: Removed LNI 192.168.204.126@tcp119 [15367.366893] LNetError: 468163:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15367.387219] LNetError: 468163:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [15367.402885] LNet: Removed LNI 192.168.204.126@tcp120 [15368.480289] LNet: Removed LNI 192.168.204.126@tcp121 [15369.504166] LNet: Removed LNI 192.168.204.126@tcp122 [15370.528787] LNet: Removed LNI 192.168.204.126@tcp123 [15371.552186] LNet: Removed LNI 192.168.204.126@tcp124 [15372.576325] LNet: Removed LNI 192.168.204.126@tcp125 [15373.600164] LNet: Removed LNI 192.168.204.126@tcp126 [15374.626449] LNet: Removed LNI 192.168.204.126@tcp127 [15375.648188] LNet: Removed LNI 192.168.204.126@tcp128 [15376.672283] LNet: Removed LNI 192.168.204.126@tcp129 [15376.677297] LNet: Removed LNI 192.168.204.126@tcp130 [15376.683116] LNet: Removed LNI 192.168.204.126@tcp131 [15376.691162] LNet: Removed LNI 192.168.204.126@tcp132 [15376.694827] LNet: Removed LNI 192.168.204.126@tcp133 [15376.697502] LNet: Removed LNI 192.168.204.126@tcp134 [15376.709496] LNet: Removed LNI 192.168.204.126@tcp135 [15377.316201] Key type .llcrypt unregistered [15377.319452] Key type ._llcrypt unregistered [15405.339350] Key type ._llcrypt registered [15405.340897] Key type .llcrypt registered [15405.468618] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [15423.477222] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [15424.637420] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15424.658600] alg: No test for adler32 (adler32-zlib) [15425.895323] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [15426.197965] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [15427.912677] Key type lgssc registered [15429.213320] Lustre: Echo OBD driver; http://www.lustre.org/ [15469.578518] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 21:37:03 (1787881023) [15493.818557] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [15497.269229] Key type lgssc unregistered [15497.639422] LNet: 472258:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15497.660371] LNetError: 472258:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15497.682709] LNet: Removed LNI 192.168.204.126@tcp [15498.629534] Key type .llcrypt unregistered [15498.634136] Key type ._llcrypt unregistered [15518.762276] Key type ._llcrypt registered [15518.764896] Key type .llcrypt registered [15518.914834] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [15520.870444] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15520.992527] alg: No test for adler32 (adler32-zlib) [15522.189624] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [15522.596268] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [15524.360194] Key type lgssc registered [15526.063944] Lustre: Echo OBD driver; http://www.lustre.org/ [15538.630883] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15538.718340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15540.212492] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15540.240712] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15540.314766] Lustre: lustre-MDT0000: new disk, initializing [15540.419445] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15540.431953] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15546.941243] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [15558.817627] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15559.014631] Lustre: 473473:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15559.079705] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15559.082950] Lustre: Skipped 1 previous similar message [15559.147594] Lustre: lustre-MDT0001: new disk, initializing [15559.230862] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15559.298861] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15559.314711] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15564.677229] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [15574.850217] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15584.116409] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15593.736074] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15594.015114] Lustre: lustre-OST0000: new disk, initializing [15594.018926] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15594.031834] Lustre: 475444:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15594.162132] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15601.599180] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [15601.817629] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15601.835565] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15601.992596] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15613.635551] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15618.462826] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing lnet_if_list [15626.541861] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [15626.672276] LNet: Added LNI 192.168.204.126@tcp10 [8/256/0/180] [15634.366494] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [15634.400934] LNet: Added LNI 192.168.204.126@tcp11 [8/256/0/180] [15640.287959] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [15640.364178] LNet: Added LNI 192.168.204.126@tcp12 [8/256/0/180] [15646.660954] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [15646.721578] LNet: Added LNI 192.168.204.126@tcp13 [8/256/0/180] [15653.311544] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [15653.352940] LNet: Added LNI 192.168.204.126@tcp14 [8/256/0/180] [15658.269927] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [15658.314648] LNet: Added LNI 192.168.204.126@tcp15 [8/256/0/180] [15663.541239] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [15663.599516] LNet: Added LNI 192.168.204.126@tcp16 [8/256/0/180] [15668.550702] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [15668.610228] LNet: Added LNI 192.168.204.126@tcp17 [8/256/0/180] [15673.501956] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [15673.549689] LNet: Added LNI 192.168.204.126@tcp18 [8/256/0/180] [15679.730957] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [15679.765756] LNet: Added LNI 192.168.204.126@tcp19 [8/256/0/180] [15685.813613] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [15685.864910] LNet: Added LNI 192.168.204.126@tcp20 [8/256/0/180] [15690.486684] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [15690.510944] LNet: Added LNI 192.168.204.126@tcp21 [8/256/0/180] [15695.037261] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [15695.071702] LNet: Added LNI 192.168.204.126@tcp22 [8/256/0/180] [15699.628394] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [15699.684192] LNet: Added LNI 192.168.204.126@tcp23 [8/256/0/180] [15704.866588] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [15704.895877] LNet: Added LNI 192.168.204.126@tcp24 [8/256/0/180] [15712.256845] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [15712.316750] LNet: Added LNI 192.168.204.126@tcp25 [8/256/0/180] [15718.916222] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [15718.960623] LNet: Added LNI 192.168.204.126@tcp26 [8/256/0/180] [15725.141732] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [15725.168425] LNet: Added LNI 192.168.204.126@tcp27 [8/256/0/180] [15732.582680] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [15732.608614] LNet: Added LNI 192.168.204.126@tcp28 [8/256/0/180] [15737.570581] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [15737.634792] LNet: Added LNI 192.168.204.126@tcp29 [8/256/0/180] [15744.203247] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [15744.228022] LNet: Added LNI 192.168.204.126@tcp30 [8/256/0/180] [15751.324844] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [15751.375646] LNet: Added LNI 192.168.204.126@tcp31 [8/256/0/180] [15758.521338] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [15758.550790] LNet: Added LNI 192.168.204.126@tcp32 [8/256/0/180] [15766.328165] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [15766.347758] LNet: Added LNI 192.168.204.126@tcp33 [8/256/0/180] [15773.130303] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [15773.159163] LNet: Added LNI 192.168.204.126@tcp34 [8/256/0/180] [15781.488564] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [15781.533557] LNet: Added LNI 192.168.204.126@tcp35 [8/256/0/180] [15790.223732] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [15790.295951] LNet: Added LNI 192.168.204.126@tcp36 [8/256/0/180] [15798.085572] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [15798.122632] LNet: Added LNI 192.168.204.126@tcp37 [8/256/0/180] [15805.721565] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [15805.773928] LNet: Added LNI 192.168.204.126@tcp38 [8/256/0/180] [15812.967596] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [15813.010851] LNet: Added LNI 192.168.204.126@tcp39 [8/256/0/180] [15820.327708] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [15820.347698] LNet: Added LNI 192.168.204.126@tcp40 [8/256/0/180] [15829.725685] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [15829.780808] LNet: Added LNI 192.168.204.126@tcp41 [8/256/0/180] [15838.710771] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [15838.744324] LNet: Added LNI 192.168.204.126@tcp42 [8/256/0/180] [15846.357806] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [15846.386703] LNet: Added LNI 192.168.204.126@tcp43 [8/256/0/180] [15854.527493] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [15854.551528] LNet: Added LNI 192.168.204.126@tcp44 [8/256/0/180] [15860.886312] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [15860.913665] LNet: Added LNI 192.168.204.126@tcp45 [8/256/0/180] [15868.603845] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [15868.640873] LNet: Added LNI 192.168.204.126@tcp46 [8/256/0/180] [15876.793192] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [15876.865904] LNet: Added LNI 192.168.204.126@tcp47 [8/256/0/180] [15885.239774] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [15885.278954] LNet: Added LNI 192.168.204.126@tcp48 [8/256/0/180] [15894.545613] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [15894.580822] LNet: Added LNI 192.168.204.126@tcp49 [8/256/0/180] [15903.127614] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [15903.164263] LNet: Added LNI 192.168.204.126@tcp50 [8/256/0/180] [15924.707218] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15924.712782] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15924.722769] Lustre: Skipped 2 previous similar messages [15924.730472] Lustre: Skipped 2 previous similar messages [15929.827309] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15929.836250] Lustre: Skipped 2 previous similar messages [15930.715618] Lustre: server umount lustre-MDT0000 complete [15934.508315] LustreError: 473466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787881491 with bad export cookie 13600745919139545592 [15934.517179] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15934.524307] LustreError: 473466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15934.855259] Lustre: server umount lustre-MDT0001 complete [15950.304155] Lustre: 472746:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787881491/real 1787881491] req@0000000012672139 x1874729190472064/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787881507 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15950.356624] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15956.353522] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [15956.450724] Lustre: 472745:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787881497/real 1787881497] req@000000005e583c1d x1874729190472192/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787881513 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15960.547963] Lustre: 472747:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787881501/real 1787881501] req@000000006a1edb85 x1874729190472448/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787881517 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15965.728202] Lustre: 472747:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787881506/real 1787881506] req@000000009c6f9613 x1874729190472576/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787881522 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15966.476762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15979.044481] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xbcbf8e5a86a80c55 [15979.066531] Lustre: MGC192.168.204.126@tcp: Connection restored to 0@lo (at 0@lo) [15979.362252] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15983.519303] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [15984.423371] LustreError: 475438:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15984.453337] LustreError: 475438:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [15989.739503] LustreError: 475437:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15993.908535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15994.361746] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15995.391585] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15999.324916] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [16004.652654] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [16008.216964] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16014.428314] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16035.834968] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16035.851119] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16035.882990] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16040.418646] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16040.432779] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16040.432836] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16041.707410] Lustre: server umount lustre-OST0000 complete [16050.661946] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16050.671000] Lustre: Skipped 1 previous similar message [16050.677158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16050.682169] Lustre: Skipped 1 previous similar message [16052.104288] Lustre: server umount lustre-MDT0000 complete [16055.777599] LustreError: 501274:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16055.798246] LustreError: 501274:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [16056.792306] LustreError: 474268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787881613 with bad export cookie 13600745919139548245 [16056.796437] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16056.810803] LustreError: 474268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [16057.109709] Lustre: server umount lustre-MDT0001 complete [16069.090468] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [16072.347571] Key type lgssc unregistered [16072.719348] LNet: 502492:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16072.739404] LNetError: 502492:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16072.744637] LNet: Removed LNI 192.168.204.126@tcp [16072.747735] LNet: Removed LNI 192.168.204.126@tcp10 [16072.750875] LNet: Removed LNI 192.168.204.126@tcp11 [16072.759100] LNet: Removed LNI 192.168.204.126@tcp12 [16072.763967] LNet: Removed LNI 192.168.204.126@tcp13 [16072.768253] LNet: Removed LNI 192.168.204.126@tcp14 [16072.771987] LNet: Removed LNI 192.168.204.126@tcp15 [16072.775438] LNet: Removed LNI 192.168.204.126@tcp16 [16072.779233] LNet: Removed LNI 192.168.204.126@tcp17 [16072.787288] LNet: Removed LNI 192.168.204.126@tcp18 [16072.796293] LNet: Removed LNI 192.168.204.126@tcp19 [16072.802840] LNet: Removed LNI 192.168.204.126@tcp20 [16072.810126] LNet: Removed LNI 192.168.204.126@tcp21 [16072.816748] LNet: Removed LNI 192.168.204.126@tcp22 [16072.823591] LNet: Removed LNI 192.168.204.126@tcp23 [16072.827877] LNet: Removed LNI 192.168.204.126@tcp24 [16072.831212] LNet: Removed LNI 192.168.204.126@tcp25 [16072.835128] LNet: Removed LNI 192.168.204.126@tcp26 [16072.839570] LNet: Removed LNI 192.168.204.126@tcp27 [16072.842816] LNet: Removed LNI 192.168.204.126@tcp28 [16072.845280] LNet: Removed LNI 192.168.204.126@tcp29 [16072.851981] LNet: Removed LNI 192.168.204.126@tcp30 [16072.860307] LNet: Removed LNI 192.168.204.126@tcp31 [16072.867248] LNet: Removed LNI 192.168.204.126@tcp32 [16072.873671] LNet: Removed LNI 192.168.204.126@tcp33 [16072.880096] LNet: Removed LNI 192.168.204.126@tcp34 [16072.889391] LNet: Removed LNI 192.168.204.126@tcp35 [16072.898199] LNet: Removed LNI 192.168.204.126@tcp36 [16072.907937] LNet: Removed LNI 192.168.204.126@tcp37 [16072.914215] LNet: Removed LNI 192.168.204.126@tcp38 [16072.916295] LNet: Removed LNI 192.168.204.126@tcp39 [16072.922928] LNet: Removed LNI 192.168.204.126@tcp40 [16072.933623] LNet: Removed LNI 192.168.204.126@tcp41 [16072.940622] LNet: Removed LNI 192.168.204.126@tcp42 [16072.945215] LNet: Removed LNI 192.168.204.126@tcp43 [16072.949204] LNet: Removed LNI 192.168.204.126@tcp44 [16072.953572] LNet: Removed LNI 192.168.204.126@tcp45 [16072.958558] LNet: Removed LNI 192.168.204.126@tcp46 [16072.966066] LNet: Removed LNI 192.168.204.126@tcp47 [16072.970623] LNet: Removed LNI 192.168.204.126@tcp48 [16072.979199] LNet: Removed LNI 192.168.204.126@tcp49 [16073.004100] LNet: Removed LNI 192.168.204.126@tcp50 [16073.807947] Key type .llcrypt unregistered [16073.814237] Key type ._llcrypt unregistered [16097.713426] Key type ._llcrypt registered [16097.717877] Key type .llcrypt registered [16097.950765] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [16115.781719] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [16117.046100] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16117.108678] alg: No test for adler32 (adler32-zlib) [16118.222697] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [16118.549558] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [16120.306701] Key type lgssc registered [16121.625574] Lustre: Echo OBD driver; http://www.lustre.org/ [16162.769995] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16162.790546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16164.126961] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16164.190082] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16164.356579] Lustre: lustre-MDT0000: new disk, initializing [16164.547769] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16164.586200] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16170.331595] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [16182.561137] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16182.699906] Lustre: 505692:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16182.799948] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16182.803470] Lustre: Skipped 1 previous similar message [16182.918489] Lustre: lustre-MDT0001: new disk, initializing [16183.123263] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16183.187900] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16183.196072] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16188.687676] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [16199.774956] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16206.543716] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16215.891900] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16216.130900] Lustre: *** cfs_fail_loc=90f, val=0*** [16216.221497] Lustre: lustre-OST0000: new disk, initializing [16216.227546] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16216.234776] Lustre: 507711:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16216.389046] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16223.823358] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16223.839274] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16224.021937] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16224.327195] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [16235.019356] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16244.720911] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16244.723351] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16244.736483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16249.856223] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16249.876129] Lustre: Skipped 2 previous similar messages [16250.770102] Lustre: server umount lustre-OST0000 complete [16260.080583] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16260.086627] LustreError: Skipped 1 previous similar message [16260.089997] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16260.110938] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16260.111577] Lustre: Skipped 2 previous similar messages [16262.979805] Lustre: server umount lustre-MDT0000 complete [16267.101760] LustreError: 505685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787881824 with bad export cookie 6748943725266217348 [16267.119672] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16267.121718] LustreError: 505685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16267.571124] Lustre: server umount lustre-MDT0001 complete [16281.366575] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [16285.003939] Key type lgssc unregistered [16285.378731] LNet: 509621:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16285.391415] LNetError: 509621:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16285.432383] LNet: Removed LNI 192.168.204.126@tcp [16286.512373] Key type .llcrypt unregistered [16286.514850] Key type ._llcrypt unregistered [16306.434194] Key type ._llcrypt registered [16306.438145] Key type .llcrypt registered [16306.552286] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [16307.582147] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16307.670388] alg: No test for adler32 (adler32-zlib) [16308.827375] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [16309.159246] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [16310.832216] Key type lgssc registered [16312.216723] Lustre: Echo OBD driver; http://www.lustre.org/ [16321.640983] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16321.680437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16323.219812] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16328.728121] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [16333.797838] LustreError: 510890:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16333.814715] LustreError: 510890:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [16338.823575] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16338.913581] LustreError: 510891:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16339.342443] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16344.912579] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [16353.327723] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16358.983225] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16366.545237] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16366.891303] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16374.500553] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [16383.692531] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16394.887698] loop0: detected capacity change from 0 to 204800000 [16395.216833] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16395.236626] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16395.271042] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16403.497956] loop0: detected capacity change from 0 to 204800000 [16403.725368] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16403.741413] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16403.756838] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16410.601798] loop0: detected capacity change from 0 to 204800000 [16417.574691] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16417.848728] Lustre: fs15246-OST0000: new disk, initializing [16417.860220] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [16417.873354] Lustre: 514488:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16417.938541] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16424.971735] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [16435.753478] loop1: detected capacity change from 0 to 204800000 [16441.930686] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16442.090524] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [16442.166315] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [16442.242487] Lustre: fs15246-MDT0000: new disk, initializing [16442.384508] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16442.437212] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16442.558559] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [16443.633323] Lustre: 515610:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000001bf66105 x1874730015573632/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787882016 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [16447.913952] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [16448.816560] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [16448.834683] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [16448.879464] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [16473.571869] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16473.590885] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [16477.668994] Lustre: server umount fs15246-MDT0000 complete [16478.702823] LustreError: 514857:0:(ldlm_lib.c:1228:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16483.813157] LustreError: 514857:0:(ldlm_lib.c:1228:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16488.116634] Lustre: server umount fs15246-OST0000 complete [16502.249175] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16502.250654] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16502.259382] LustreError: Skipped 1 previous similar message [16502.263395] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16502.282659] Lustre: Skipped 1 previous similar message [16502.308356] Lustre: Skipped 1 previous similar message [16504.298670] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16504.310790] Lustre: Skipped 1 previous similar message [16507.425744] Lustre: server umount lustre-OST0000 complete [16513.507312] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16513.512547] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16513.534842] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16519.649260] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16519.659335] Lustre: Skipped 3 previous similar messages [16526.304530] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16526.629803] Lustre: server umount lustre-MDT0000 complete [16529.906152] LustreError: 510891:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16529.939250] LustreError: 510891:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [16531.493177] LustreError: 511627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787882088 with bad export cookie 13524380448113626895 [16531.497355] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16531.505903] LustreError: 511627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [16531.947965] Lustre: server umount lustre-MDT0001 complete [16544.107279] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [16547.332500] Key type lgssc unregistered [16547.658530] LNet: 518711:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16547.669349] LNetError: 518711:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16547.686449] LNet: Removed LNI 192.168.204.126@tcp [16548.711061] Key type .llcrypt unregistered [16548.713460] Key type ._llcrypt unregistered [16567.915957] Key type ._llcrypt registered [16567.918900] Key type .llcrypt registered [16568.151213] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [16569.512099] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16569.634645] alg: No test for adler32 (adler32-zlib) [16570.781449] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [16571.168318] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [16572.896262] Key type lgssc registered [16574.608066] Lustre: Echo OBD driver; http://www.lustre.org/ [16612.100973] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [16621.825351] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [16662.992750] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16663.041381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16664.373727] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16664.439972] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16664.541611] Lustre: lustre-MDT0000: new disk, initializing [16664.616605] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16664.641150] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16668.888456] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [16679.099186] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16679.270919] Lustre: 523182:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16679.312105] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16679.316571] Lustre: Skipped 1 previous similar message [16679.451501] Lustre: lustre-MDT0001: new disk, initializing [16679.559666] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16679.598340] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16679.616543] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16684.323471] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [16694.530699] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16700.563249] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16707.283374] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16707.566618] Lustre: lustre-OST0000: new disk, initializing [16707.571460] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16707.576749] Lustre: 525151:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16707.676709] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16714.447920] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [16716.906097] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16716.915073] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16717.007605] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16724.277949] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16728.624976] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [16728.902386] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [16733.732969] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [16733.944944] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [16737.763533] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16737.765981] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16737.777512] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16742.881765] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16742.891704] Lustre: Skipped 2 previous similar messages [16748.027472] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16748.039538] Lustre: Skipped 1 previous similar message [16750.048613] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16750.191288] Lustre: server umount lustre-OST0000 complete [16756.194443] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16756.207633] Lustre: Skipped 1 previous similar message [16756.211203] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16756.217260] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16756.224877] Lustre: Skipped 1 previous similar message [16760.853187] Lustre: server umount lustre-MDT0000 complete [16764.798309] LustreError: 523175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787882321 with bad export cookie 42532310260034218 [16764.810598] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16764.813397] LustreError: 523175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [16765.217649] Lustre: server umount lustre-MDT0001 complete [16791.908649] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [16806.785565] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [16844.509964] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 16583 sec ========= 21:59:59 (1787882399) [16846.639987] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 22:00:01 (1787882401) === [16850.751458] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 22:00:05 (1787882405) === [16894.331813] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [16897.116829] Key type lgssc unregistered [16897.370275] LNet: 531816:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16897.375834] LNetError: 531816:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16897.388480] LNet: Removed LNI 192.168.204.126@tcp [16898.131609] Key type .llcrypt unregistered [16898.134128] Key type ._llcrypt unregistered