[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 495837558 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2480MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524588K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001017] APIC: Switch to symmetric I/O mode setup [ 0.003395] x2apic enabled [ 0.004013] Switched APIC routing to physical x2apic. [ 0.006014] kvm-guest: setup PV IPIs [ 0.009000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009031] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010021] pid_max: default: 32768 minimum: 301 [ 0.011190] LSM: Security Framework initializing [ 0.012084] Yama: becoming mindful. [ 0.013058] SELinux: Initializing. [ 0.014097] *** VALIDATE selinux *** [ 0.022644] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027762] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028185] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029132] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030176] *** VALIDATE tmpfs *** [ 0.031535] *** VALIDATE proc *** [ 0.033256] *** VALIDATE cgroup *** [ 0.034020] *** VALIDATE cgroup2 *** [ 0.035323] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036188] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037013] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038042] Spectre V2 : User space: Vulnerable [ 0.039013] Speculative Store Bypass: Vulnerable [ 0.042258] debug: unmapping init [mem 0xffffffffaac59000-0xffffffffaac60fff] [ 0.045215] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046758] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047034] ... version: 2 [ 0.048014] ... bit width: 48 [ 0.049013] ... generic registers: 4 [ 0.050018] ... value mask: 0000ffffffffffff [ 0.051023] ... max period: 00007fffffffffff [ 0.052023] ... fixed-purpose events: 3 [ 0.053019] ... event mask: 000000070000000f [ 0.055298] rcu: Hierarchical SRCU implementation. [ 0.057733] smp: Bringing up secondary CPUs ... [ 0.058706] x86: Booting SMP configuration: [ 0.059047] .... node #0, CPUs: #1 #2 #3 [ 0.063154] smp: Brought up 1 node, 4 CPUs [ 0.065015] smpboot: Max logical packages: 1 [ 0.066012] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.138782] node 0 deferred pages initialised in 70ms [ 0.143163] devtmpfs: initialized [ 0.144307] x86/mm: Memory block size: 128MB [ 0.146000] gcov: version magic: 0x41383552 [ 0.149301] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.150084] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.152356] pinctrl core: initialized pinctrl subsystem [ 0.155235] [ 0.155898] ************************************************************* [ 0.158021] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.161018] ** ** [ 0.164018] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.166014] ** ** [ 0.168018] ** This means that this kernel is built to expose internal ** [ 0.171017] ** IOMMU data structures, which may compromise security on ** [ 0.173015] ** your system. ** [ 0.176019] ** ** [ 0.178016] ** If you see this message and you are not debugging the ** [ 0.180024] ** kernel, report this immediately to your vendor! ** [ 0.183014] ** ** [ 0.185014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.187016] ************************************************************* [ 0.189880] NET: Registered protocol family 16 [ 0.191381] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.193057] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.195070] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.198527] cpuidle: using governor menu [ 0.200592] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.202526] PCI: Using configuration type 1 for base access [ 0.204162] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.212116] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.214035] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.217037] cryptd: max_cpu_qlen set to 1000 [ 0.220249] ACPI: Added _OSI(Module Device) [ 0.222024] ACPI: Added _OSI(Processor Device) [ 0.224015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.225017] ACPI: Added _OSI(Processor Aggregator Device) [ 0.230736] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.237509] ACPI: Interpreter enabled [ 0.239065] ACPI: PM: (supports S0 S3 S4 S5) [ 0.240019] ACPI: Using IOAPIC for interrupt routing [ 0.241117] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.244338] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.254492] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.257057] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.260028] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.264102] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.269650] acpiphp: Slot [2] registered [ 0.271296] acpiphp: Slot [5] registered [ 0.273171] acpiphp: Slot [6] registered [ 0.275175] acpiphp: Slot [7] registered [ 0.277194] acpiphp: Slot [8] registered [ 0.278267] acpiphp: Slot [9] registered [ 0.280287] acpiphp: Slot [10] registered [ 0.282259] acpiphp: Slot [3] registered [ 0.284135] acpiphp: Slot [4] registered [ 0.285195] acpiphp: Slot [11] registered [ 0.287144] acpiphp: Slot [12] registered [ 0.289159] acpiphp: Slot [13] registered [ 0.290130] acpiphp: Slot [14] registered [ 0.292173] acpiphp: Slot [15] registered [ 0.294129] acpiphp: Slot [16] registered [ 0.296143] acpiphp: Slot [17] registered [ 0.297087] acpiphp: Slot [18] registered [ 0.298217] acpiphp: Slot [19] registered [ 0.300122] acpiphp: Slot [20] registered [ 0.302135] acpiphp: Slot [21] registered [ 0.303212] acpiphp: Slot [22] registered [ 0.304113] acpiphp: Slot [23] registered [ 0.306124] acpiphp: Slot [24] registered [ 0.307234] acpiphp: Slot [25] registered [ 0.309159] acpiphp: Slot [26] registered [ 0.311127] acpiphp: Slot [27] registered [ 0.313147] acpiphp: Slot [28] registered [ 0.314138] acpiphp: Slot [29] registered [ 0.316125] acpiphp: Slot [30] registered [ 0.318197] acpiphp: Slot [31] registered [ 0.320134] PCI host bridge to bus 0000:00 [ 0.321025] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.324035] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.326030] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.329060] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.332016] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.333037] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.335226] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.338133] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.341542] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.352021] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.357727] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.362033] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.364020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.367024] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.369310] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.370575] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.374053] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.377887] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.384019] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.396019] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.401871] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.406837] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.413018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.417016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.434031] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.444734] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.451035] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.457023] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.475019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.484765] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.491020] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.497024] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.516029] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.524539] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.532020] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.538019] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.554021] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.565769] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.573026] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.581021] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.595025] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.605659] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.619020] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.633027] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.646012] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.655365] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.658462] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.661439] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.663431] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.666383] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.670140] iommu: Default domain type: Passthrough [ 0.672602] SCSI subsystem initialized [ 0.674152] ACPI: bus type USB registered [ 0.675113] usbcore: registered new interface driver usbfs [ 0.676156] usbcore: registered new interface driver hub [ 0.678105] usbcore: registered new device driver usb [ 0.680165] pps_core: LinuxPPS API ver. 1 registered [ 0.681009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.682080] PTP clock support registered [ 0.684066] EDAC MC: Ver: 3.0.0 [ 0.686167] PCI: Using ACPI for IRQ routing [ 0.687519] NetLabel: Initializing [ 0.688013] NetLabel: domain hash size = 128 [ 0.690014] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.691087] NetLabel: unlabeled traffic allowed by default [ 0.693039] vgaarb: loaded [ 0.694262] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.695013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.705000] clocksource: Switched to clocksource kvm-clock [ 0.807065] VFS: Disk quotas dquot_6.6.0 [ 0.808866] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.811466] *** VALIDATE ramfs *** [ 0.812556] *** VALIDATE hugetlbfs *** [ 0.814308] pnp: PnP ACPI init [ 0.816737] pnp: PnP ACPI: found 6 devices [ 0.831343] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.833948] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.835758] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.837599] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.839922] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.842329] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.845037] NET: Registered protocol family 2 [ 0.847082] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.851317] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.854129] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.859696] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.862606] TCP: Hash tables configured (established 65536 bind 65536) [ 0.865591] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.868792] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.871586] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.874502] NET: Registered protocol family 1 [ 0.877556] RPC: Registered named UNIX socket transport module. [ 0.879281] RPC: Registered udp transport module. [ 0.880892] RPC: Registered tcp transport module. [ 0.882192] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.884363] NET: Registered protocol family 44 [ 0.885815] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.887717] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.889243] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.891175] PCI: CLS 0 bytes, default 64 [ 0.893496] Unpacking initramfs... [ 2.281862] debug: unmapping init [mem 0xffff8d94bcc54000-0xffff8d94bffbffff] [ 2.284896] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.286618] software IO TLB: mapped [mem 0x00000000b8c54000-0x00000000bcc54000] (64MB) [ 2.290049] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.803387] Initialise system trusted keyrings [ 2.805026] Key type blacklist registered [ 2.806631] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.813859] zbud: loaded [ 2.816419] *** VALIDATE nfs *** [ 2.817174] *** VALIDATE nfs4 *** [ 2.818616] pstore: using deflate compression [ 2.821717] Platform Keyring initialized [ 2.917963] NET: Registered protocol family 38 [ 2.919819] Key type asymmetric registered [ 2.921569] Asymmetric key parser 'x509' registered [ 2.923958] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.927246] io scheduler mq-deadline registered [ 2.928958] io scheduler kyber registered [ 2.930771] io scheduler bfq registered [ 2.932088] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.933871] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.936554] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.939628] ACPI: Power Button [PWRF] [ 2.947136] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.955320] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.968474] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.974917] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.990152] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.017303] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.043164] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.046518] Non-volatile memory driver v1.3 [ 3.047610] Linux agpgart interface v0.103 [ 3.081134] virtio_blk virtio1: [vda] 145896 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.084708] vda: detected capacity change from 0 to 74698752 [ 3.101260] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.104371] vdb: detected capacity change from 0 to 1073741824 [ 3.119051] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.121309] vdc: detected capacity change from 0 to 2621440000 [ 3.134880] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.137232] vdd: detected capacity change from 0 to 2621440000 [ 3.148692] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.151740] vde: detected capacity change from 0 to 4294967296 [ 3.164098] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.166900] vdf: detected capacity change from 0 to 4294967296 [ 3.174468] libphy: Fixed MDIO Bus: probed [ 3.180728] usbcore: registered new interface driver usbserial_generic [ 3.183576] usbserial: USB Serial support registered for generic [ 3.186431] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.190481] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.191709] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.193334] mousedev: PS/2 mouse device common for all mice [ 3.195478] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.198216] rtc_cmos 00:05: RTC can wake from S4 [ 3.200795] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.202705] rtc_cmos 00:05: registered as rtc0 [ 3.205736] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.206037] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.210282] intel_pstate: CPU model not supported [ 3.212624] hid: raw HID events driver (C) Jiri Kosina [ 3.214257] usbcore: registered new interface driver usbhid [ 3.215571] usbhid: USB HID core driver [ 3.216676] drop_monitor: Initializing network drop monitor service [ 3.218772] Initializing XFRM netlink socket [ 3.220130] NET: Registered protocol family 10 [ 3.222918] Segment Routing with IPv6 [ 3.223907] NET: Registered protocol family 17 [ 3.225679] mpls_gso: MPLS GSO support [ 3.231329] RAS: Correctable Errors collector initialized. [ 3.233415] AVX version of gcm_enc/dec engaged. [ 3.235198] AES CTR mode by8 optimization enabled [ 3.303458] sched_clock: Marking stable (3303425460, 0)->(4242884595, -939459135) [ 3.306952] registered taskstats version 1 [ 3.309487] Loading compiled-in X.509 certificates [ 3.311217] zswap: loaded using pool lzo/zbud [ 3.329710] Key type big_key registered [ 3.339101] Key type encrypted registered [ 3.340822] ima: No TPM chip found, activating TPM-bypass! [ 3.342993] ima: Allocated hash algorithm: sha1 [ 3.344835] ima: No architecture policies found [ 3.346559] evm: Initialising EVM extended attributes: [ 3.348302] evm: security.selinux [ 3.349632] evm: security.ima [ 3.350720] evm: security.capability [ 3.351926] evm: HMAC attrs: 0x1 [ 3.354313] rtc_cmos 00:05: setting system clock to 2026-09-03 13:02:10 UTC (1788440530) [ 3.361560] debug: unmapping init [mem 0xffffffffabc03000-0xffffffffabdfffff] [ 3.364405] debug: unmapping init [mem 0xffffffffaa982000-0xffffffffaac58fff] [ 3.373141] Write protecting the kernel read-only data: 28672k [ 3.375988] debug: unmapping init [mem 0xffffffffa9003000-0xffffffffa91fffff] [ 3.378212] debug: unmapping init [mem 0xffffffffa9914000-0xffffffffa99fffff] [ 3.417258] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.425279] systemd[1]: Detected virtualization kvm. [ 3.427120] systemd[1]: Detected architecture x86-64. [ 3.428943] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.454291] systemd[1]: No hostname configured. [ 3.456072] systemd[1]: Set hostname to . [ 3.458376] random: systemd: uninitialized urandom read (16 bytes read) [ 3.461048] systemd[1]: Initializing machine ID from random generator. [ 3.519097] random: ln: uninitialized urandom read (6 bytes read) [ 3.591366] random: systemd: uninitialized urandom read (16 bytes read) [ 3.595659] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.604467] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.613937] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Reached target Timers. [ OK ] Reached target Local File Systems. [ OK ] Reached target Initrd Root Device. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Slices. [ OK ] Reached target Swap. Starting Create Volatile Files and Directories... [ OK ] Reached target Local Encrypted Volumes. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Reached target Paths. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.179869] device-mapper: uevent: version 1.0.3 [ 5.186788] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 7.388670] virtio_net virtio0 ens2: renamed from eth0 [ 7.394155] random: fast init done [ 7.692398] scsi host0: ata_piix [ 7.950347] scsi host1: ata_piix [ 7.959916] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 7.970532] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 12.865793] random: crng init done [ 12.869263] random: 7 urandom warning(s) missed due to ratelimiting [ 15.211508] dracut-initqueue[591]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ 16.370669] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Reached target Remote File Systems. [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 18.822208] printk: systemd: 26 output lines suppressed due to ratelimiting [ 19.491681] SELinux: Disabled at runtime. [ 19.627926] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 19.644166] systemd[1]: Detected virtualization kvm. [ 19.650125] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 21.243586] systemd[1]: initrd-switch-root.service: Succeeded. [ 21.249474] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 21.256922] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 21.263354] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 21.270490] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 21.295576] systemd[1]: Starting Journal Service... Starting Journal Service... [ 21.308649] systemd[1]: Created slice system-getty.slice. [ OK ] Created slice system-getty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Switch Root. Starting Create list of required st…ce nodes for the current kernel... Mounting Kernel Debug File System... [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on RPCbind Server Activation Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Mounting POSIX Message Queue File System... [ OK ] Created slice system-serial\x2dgetty.slice. Activating swap /dev/disk/by-label/SWAP... Mounting Huge Pages File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ 21.798471] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 22.685673] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 23.649682] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 23.900383] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 24.253267] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 24.337192] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit)[ 28.608575] Key type dns_resolver registered [** ] A start job is running for Configur…-only root support (7s / no limit)[ 29.431873] NFS: Registering the id_resolver key type [ 29.437952] Key type id_resolver registered [*[0[ 29.440584] Key type id_legacy registered ;1;31m** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (8s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg113-server login: [ 58.122718] hrtimer: interrupt took 7777695 ns [ 84.732899] libcfs: loading out-of-tree module taints kernel. [ 84.822307] Key type ._llcrypt registered [ 84.826289] Key type .llcrypt registered [ 84.907929] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [ 103.516940] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 105.006310] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 105.029817] alg: No test for adler32 (adler32-zlib) [ 106.382224] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 107.201978] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 108.975183] Key type lgssc registered [ 110.808801] Lustre: Echo OBD driver; http://www.lustre.org/ [ 131.301439] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 173.504756] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 187.122745] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 187.150276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 188.510534] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 188.534285] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 188.647482] Lustre: lustre-MDT0000: new disk, initializing [ 188.748568] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 188.777628] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 193.370625] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 207.603885] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 207.736113] Lustre: 6517:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 207.785135] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 207.794070] Lustre: Skipped 1 previous similar message [ 207.948139] Lustre: lustre-MDT0001: new disk, initializing [ 208.015759] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 208.064831] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 208.085348] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 212.492375] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 217.809257] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 228.132240] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 228.425738] Lustre: lustre-OST0000: new disk, initializing [ 228.430570] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 228.439509] Lustre: 8455:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 228.499102] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 229.674071] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 229.692654] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 229.830920] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 235.809603] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 249.707896] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 249.849143] Lustre: lustre-OST0001: new disk, initializing [ 249.852153] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 249.860254] Lustre: 9527:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 249.943689] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 255.296984] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 256.005314] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 256.016870] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 256.096108] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 266.708633] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 278.268686] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 285.444724] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing check_logdir /tmp/testlogs/ [ 290.829409] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing yml_node [ 295.677465] Lustre: DEBUG MARKER: Client: 2.17.56.51 [ 298.637850] Lustre: DEBUG MARKER: MDS: 2.17.56.51 [ 301.531322] Lustre: DEBUG MARKER: OSS: 2.17.56.51 [ 303.121273] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 3 09:07:09 EDT 2026 [ 320.752638] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 322.309966] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 337.892255] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 337.894163] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 337.913634] Lustre: Skipped 2 previous similar messages [ 337.932727] Lustre: Skipped 3 previous similar messages [ 343.010191] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 343.018094] Lustre: Skipped 3 previous similar messages [ 343.741413] Lustre: server umount lustre-MDT0000 complete [ 347.407525] LustreError: 6509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440874 with bad export cookie 11104373308040357201 [ 347.415987] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 347.425274] LustreError: 6509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 348.129929] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 348.134802] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 348.142881] Lustre: Skipped 2 previous similar messages [ 348.158042] Lustre: Skipped 1 previous similar message [ 352.160294] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 352.165349] Lustre: Skipped 1 previous similar message [ 353.874567] Lustre: server umount lustre-MDT0001 complete [ 358.025832] Lustre: server umount lustre-OST0000 complete [ 362.378239] Lustre: server umount lustre-OST0001 complete [ 370.603550] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [ 380.576717] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 423.602281] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 423.752718] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 423.768635] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 423.804148] Lustre: lustre-MDT0000: new disk, initializing [ 423.869258] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 423.879828] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 428.206684] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 440.212798] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 440.339877] Lustre: 15084:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 440.418122] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 440.422281] Lustre: Skipped 1 previous similar message [ 440.551290] Lustre: lustre-MDT0001: new disk, initializing [ 440.647894] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 440.696786] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 440.711617] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 445.772548] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 455.726732] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 462.718966] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 469.547273] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 469.974113] Lustre: lustre-OST0000: new disk, initializing [ 469.989936] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 470.008782] Lustre: 17050:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 470.146819] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 471.201048] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 471.223691] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 471.426665] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 476.485951] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 486.653488] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 491.444562] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 491.776539] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 497.127489] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 497.445867] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 502.240737] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 502.249110] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 502.258041] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 502.289083] Lustre: Skipped 1 previous similar message [ 512.488826] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 512.505565] Lustre: Skipped 4 previous similar messages [ 514.015960] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 514.141780] Lustre: server umount lustre-OST0000 complete [ 522.720362] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 522.744990] Lustre: Skipped 1 previous similar message [ 525.240740] Lustre: server umount lustre-MDT0000 complete [ 527.846939] LustreError: 15094:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 527.869543] LustreError: 15094:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 529.802267] LustreError: 15076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441056 with bad export cookie 11104373308040358734 [ 529.803360] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 529.818356] LustreError: 15076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 530.183255] Lustre: server umount lustre-MDT0001 complete [ 535.695607] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 09:11:01 (1788441061) [ 539.273154] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 541.409744] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 09:11:07 (1788441067) [ 548.871623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 549.266030] LustreError: 19463:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 549.388856] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 554.234296] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 554.475972] LustreError: 19464:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 559.586543] LustreError: 19463:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 563.427102] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 568.608373] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 577.405849] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 583.664800] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 590.371725] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 596.925948] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 604.397098] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 615.392670] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 615.398585] LustreError: Skipped 1 previous similar message [ 615.402469] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 615.423499] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 615.435964] Lustre: Skipped 2 previous similar messages [ 618.467188] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 628.191244] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 628.371641] Lustre: server umount lustre-OST0000 complete [ 633.825517] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 633.841712] Lustre: Skipped 1 previous similar message [ 638.423566] Lustre: server umount lustre-MDT0000 complete [ 638.950144] LustreError: 19459:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 638.972957] LustreError: 19459:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 643.009592] LustreError: 19445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441170 with bad export cookie 11104373308040359616 [ 643.018843] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 643.023679] LustreError: 19445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 643.786917] Lustre: server umount lustre-MDT0001 complete [ 659.236112] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 663.848835] Key type lgssc unregistered [ 664.364972] LNet: 22999:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 664.372104] LNetError: 22999:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 664.387880] LNet: Removed LNI 192.168.201.113@tcp [ 665.851262] Key type .llcrypt unregistered [ 665.853891] Key type ._llcrypt unregistered [ 690.952071] Key type ._llcrypt registered [ 690.955121] Key type .llcrypt registered [ 691.123904] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 692.874714] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 692.941417] alg: No test for adler32 (adler32-zlib) [ 694.200621] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 694.592738] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 696.479827] Key type lgssc registered [ 698.394679] Lustre: Echo OBD driver; http://www.lustre.org/ [ 708.591977] loop: module loaded [ 708.699435] loop0: detected capacity change from 0 to 204800000 [ 709.039526] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 709.048647] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 709.399305] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 715.400202] loop0: detected capacity change from 0 to 204800000 [ 715.608634] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 715.629022] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 715.663544] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 720.755844] loop0: detected capacity change from 0 to 204800000 [ 727.494338] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 727.517152] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 728.874309] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 728.931727] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 729.042465] Lustre: lustre-MDT0000: new disk, initializing [ 729.120532] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 729.137218] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 733.379978] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 743.219431] loop1: detected capacity change from 0 to 204800000 [ 749.680133] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 750.152592] Lustre: lustre-OST07c6: new disk, initializing [ 750.156744] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 750.163345] Lustre: Skipped 1 previous similar message [ 750.167871] Lustre: 26188:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 750.233732] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 757.235371] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 760.420708] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 760.427690] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 760.457373] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 762.663493] Lustre: 25176:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 762.679988] Lustre: 25176:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 762.690734] Lustre: 25176:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 762.702168] Lustre: 25176:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 762.711291] Lustre: 25176:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 762.720561] Lustre: 25176:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 780.701618] Lustre: 26196:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 780.708411] Lustre: 26196:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 780.713718] Lustre: 26196:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 780.723990] Lustre: 26196:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 780.733764] Lustre: 26196:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 780.740142] Lustre: 26196:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 785.667314] Lustre: 25176:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 785.679915] Lustre: 25176:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 785.687046] Lustre: 25176:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 785.694500] Lustre: 25176:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 785.711527] Lustre: 25176:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 785.717507] Lustre: 25176:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 785.722365] Lustre: 25176:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 785.727257] Lustre: 25176:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 785.731590] Lustre: 25176:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 785.735362] Lustre: 25176:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 785.741523] Lustre: 25176:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 785.747966] Lustre: 25176:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 789.010503] Lustre: 25175:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 789.027790] Lustre: 25175:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 789.047736] Lustre: 25175:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 789.061639] Lustre: 25175:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 789.074994] Lustre: 25175:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 789.078271] Lustre: 25175:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 789.090516] Lustre: 25175:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 789.103436] Lustre: 25175:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 789.115864] Lustre: 25175:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 789.128836] Lustre: 25175:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 789.135859] Lustre: 25175:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 789.145669] Lustre: 25175:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 794.394716] Lustre: 25175:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 794.403114] Lustre: 25175:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 794.413303] Lustre: 25175:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 794.424468] Lustre: 25175:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 794.440635] Lustre: 25175:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 794.462166] Lustre: 25175:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 794.472137] Lustre: 25175:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 794.477922] Lustre: 25175:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 794.482951] Lustre: 25175:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 794.488670] Lustre: 25175:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 794.493600] Lustre: 25175:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 794.499159] Lustre: 25175:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 803.186399] Lustre: 25176:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 803.196458] Lustre: 25176:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 803.211118] Lustre: 25176:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 803.225606] Lustre: 25176:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 803.237444] Lustre: 25176:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 803.248678] Lustre: 25176:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 803.256676] Lustre: 25176:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 803.267451] Lustre: 25176:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 803.277065] Lustre: 25176:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 803.285963] Lustre: 25176:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 803.294962] Lustre: 25176:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 803.306234] Lustre: 25176:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 857.570813] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 857.580887] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 857.614248] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 862.694926] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 867.809373] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 869.855165] Lustre: lustre-OST07c6 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 870.032844] Lustre: server umount lustre-OST07c6 complete [ 879.975192] Lustre: server umount lustre-MDT0000 complete [ 897.943116] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 900.835035] Key type lgssc unregistered [ 901.121708] LNet: 28751:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 901.134475] LNetError: 28751:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 901.151566] LNet: Removed LNI 192.168.201.113@tcp [ 901.920218] Key type .llcrypt unregistered [ 901.922087] Key type ._llcrypt unregistered [ 923.225229] Key type ._llcrypt registered [ 923.227794] Key type .llcrypt registered [ 923.314682] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 924.760489] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 924.806584] alg: No test for adler32 (adler32-zlib) [ 925.921472] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 926.261367] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 927.943621] Key type lgssc registered [ 929.506204] Lustre: Echo OBD driver; http://www.lustre.org/ [ 939.829256] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 939.884088] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 941.389178] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 947.410518] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 951.788448] LustreError: 30020:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 951.814750] LustreError: 30020:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 956.897075] LustreError: 30021:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 958.915617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 959.596535] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 966.593120] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 976.007453] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 983.628674] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 992.195491] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 992.635570] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 999.726653] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 1008.233651] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1011.066100] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 1029.599962] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1029.603408] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1029.620424] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1030.942266] Lustre: server umount lustre-OST0000 complete [ 1036.256626] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1036.273653] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1036.288148] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1041.890512] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1041.906956] Lustre: Skipped 2 previous similar messages [ 1044.058155] Lustre: server umount lustre-MDT0000 complete [ 1045.987588] LustreError: 30021:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1045.998414] LustreError: 30021:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1048.824227] LustreError: 30001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441575 with bad export cookie 18224866502929915668 [ 1048.833774] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1048.834403] LustreError: 30001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1049.191795] Lustre: server umount lustre-MDT0001 complete [ 1062.245123] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 1065.499574] Key type lgssc unregistered [ 1065.872776] LNet: 33765:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1065.894210] LNetError: 33765:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1066.929277] LNet: Removed LNI 192.168.201.113@tcp [ 1067.763105] Key type .llcrypt unregistered [ 1067.765985] Key type ._llcrypt unregistered [ 1087.410858] Key type ._llcrypt registered [ 1087.416950] Key type .llcrypt registered [ 1087.551535] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 1088.798463] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1088.918147] alg: No test for adler32 (adler32-zlib) [ 1089.989864] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1090.266049] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 1091.983213] Key type lgssc registered [ 1093.451261] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1103.166611] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1103.193943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1104.831339] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1109.752383] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 1114.083446] LustreError: 35038:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1114.100208] LustreError: 35038:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1119.219662] LustreError: 35039:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1119.480151] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1119.827620] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1123.497213] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 1131.943284] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1138.598620] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1145.810953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1146.104824] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1149.203323] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1153.389501] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 1162.429311] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1179.622172] Lustre: server umount lustre-OST0000 complete [ 1185.250132] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1185.261345] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1185.281146] Lustre: Skipped 1 previous similar message [ 1190.371973] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1190.387136] Lustre: Skipped 2 previous similar messages [ 1193.521444] Lustre: server umount lustre-MDT0000 complete [ 1195.493932] LustreError: 35039:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1195.533403] LustreError: 35039:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1195.983252] Lustre: 34348:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788441707/real 1788441707] req@00000000d0ff52ce x1875316957016832/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788441723 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1196.072179] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1198.877592] LustreError: 35020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441726 with bad export cookie 9161434304289926759 [ 1198.888752] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1198.903108] LustreError: 35020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1199.537597] Lustre: server umount lustre-MDT0001 complete [ 1213.400157] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 1216.357969] Key type lgssc unregistered [ 1216.727497] LNet: 38597:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1216.737415] LNetError: 38597:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1216.762764] LNet: Removed LNI 192.168.201.113@tcp [ 1217.518589] Key type .llcrypt unregistered [ 1217.520695] Key type ._llcrypt unregistered [ 1245.742457] Key type ._llcrypt registered [ 1245.748713] Key type .llcrypt registered [ 1245.897916] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 1247.023694] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1247.133108] alg: No test for adler32 (adler32-zlib) [ 1248.280313] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1248.620996] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 1250.303192] Key type lgssc registered [ 1251.551464] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1260.057429] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1260.078103] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1261.757589] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1266.346245] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 1271.267227] LustreError: 40013:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1271.295888] LustreError: 40013:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1276.404862] LustreError: 40014:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1277.712564] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1278.261970] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1282.969912] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 1292.788823] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1299.971750] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1308.405893] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1309.018500] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1317.191570] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 1325.847104] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1331.570756] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1336.800790] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1336.807367] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1336.825185] Lustre: Skipped 1 previous similar message [ 1336.837062] Lustre: Skipped 2 previous similar messages [ 1337.327206] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.13@tcp (stopping) [ 1338.888414] Lustre: server umount lustre-MDT0000 complete [ 1341.924071] LustreError: 40007:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1341.968384] LustreError: 40007:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1343.076051] LustreError: 39994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441870 with bad export cookie 15461645827636805170 [ 1343.079868] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1343.084503] LustreError: 39994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1347.040152] LustreError: 40009:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1347.044573] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1347.069959] LustreError: 40009:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1347.107207] Lustre: Skipped 1 previous similar message [ 1347.116485] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1349.501776] Lustre: server umount lustre-MDT0001 complete [ 1371.066378] Lustre: server umount lustre-OST0000 complete [ 1388.552287] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 1391.571991] Key type lgssc unregistered [ 1391.895526] LNet: 43882:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1391.900965] LNetError: 43882:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1391.920258] LNet: Removed LNI 192.168.201.113@tcp [ 1392.874461] Key type .llcrypt unregistered [ 1392.877141] Key type ._llcrypt unregistered [ 1421.220607] Key type ._llcrypt registered [ 1421.222727] Key type .llcrypt registered [ 1421.315239] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 1422.700475] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1422.739865] alg: No test for adler32 (adler32-zlib) [ 1423.767865] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1424.033947] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 1425.735874] Key type lgssc registered [ 1427.098517] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1435.856921] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1435.885667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1437.489077] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1442.588795] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 1446.882099] LustreError: 45302:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1446.911854] LustreError: 45302:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1452.018471] LustreError: 45303:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1452.192177] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1452.831431] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1458.051264] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 1466.234608] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1473.407946] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1481.320858] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1481.738863] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1482.742176] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1487.603909] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 1496.075673] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1503.202544] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1503.214289] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1503.228557] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1504.200758] Lustre: lustre-OST0000: Not available for connect from 192.168.201.13@tcp (stopping) [ 1507.313430] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1507.321212] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1507.350850] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1509.202915] Lustre: server umount lustre-OST0000 complete [ 1529.313260] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1529.325509] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1529.344547] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1529.350364] Lustre: Skipped 1 previous similar message [ 1532.908414] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1534.371492] Lustre: server umount lustre-MDT0000 complete [ 1538.024253] LustreError: 45303:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1538.050193] LustreError: 45303:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1539.020130] LustreError: 45283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442066 with bad export cookie 14320516816760336635 [ 1539.037188] LustreError: 45283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1539.038327] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1545.781842] Lustre: server umount lustre-MDT0001 complete [ 1559.700480] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 1563.721608] Key type lgssc unregistered [ 1564.191602] LNet: 49013:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1564.209647] LNetError: 49013:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1564.237877] LNet: Removed LNI 192.168.201.113@tcp [ 1565.734322] Key type .llcrypt unregistered [ 1565.736675] Key type ._llcrypt unregistered [ 1592.827417] Key type ._llcrypt registered [ 1592.828876] Key type .llcrypt registered [ 1592.907948] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 1594.220983] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1594.280541] alg: No test for adler32 (adler32-zlib) [ 1595.888455] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1596.287322] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 1598.047290] Key type lgssc registered [ 1599.722411] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1609.701891] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1609.751300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1611.422031] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1616.349215] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 1620.989170] LustreError: 50434:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1621.022230] LustreError: 50434:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1626.093352] LustreError: 50433:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1627.476811] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1628.317792] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1634.143540] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 1642.336225] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1649.117636] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1657.974365] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1658.448087] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1664.746838] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 1673.547594] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1681.018359] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1689.355501] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1693.253703] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1706.535559] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-03 9h30m31s [ 1708.541381] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1712.111955] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1712.112601] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1712.132910] Lustre: Skipped 1 previous similar message [ 1712.146132] Lustre: Skipped 1 previous similar message [ 1713.207986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1713.212041] Lustre: Skipped 3 previous similar messages [ 1715.140133] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.13@tcp (stopping) [ 1717.431605] Lustre: server umount lustre-MDT0000 complete [ 1719.276029] LustreError: 50433:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1719.292454] LustreError: 50433:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1721.165497] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1722.287259] LustreError: 50430:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1728.444421] LustreError: 51572:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1728.474235] LustreError: 51572:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1728.574428] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1728.704400] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1728.975156] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1729.036869] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1730.055644] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1733.927781] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 1738.022680] Lustre: DEBUG MARKER: Wait for df (36759) ... [ 1740.054918] Lustre: DEBUG MARKER: done [ 1745.387692] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1745.423237] Lustre: Skipped 1 previous similar message [ 1745.441521] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1745.458280] Lustre: Skipped 2 previous similar messages [ 1750.503046] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1759.199532] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1759.510279] Lustre: server umount lustre-OST0000 complete [ 1765.862554] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1765.863593] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1765.877252] Lustre: Skipped 1 previous similar message [ 1765.880482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1765.895200] Lustre: Skipped 4 previous similar messages [ 1769.662743] Lustre: server umount lustre-MDT0000 complete [ 1773.036573] LustreError: 50433:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1773.051384] LustreError: 50433:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1773.173255] LustreError: 50413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442300 with bad export cookie 16882627699758654837 [ 1773.182761] LustreError: 50413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1773.184242] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1773.543377] Lustre: server umount lustre-MDT0001 complete [ 1784.368491] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 1787.078964] Key type lgssc unregistered [ 1787.324676] LNet: 55387:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1787.333876] LNetError: 55387:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1787.350878] LNet: Removed LNI 192.168.201.113@tcp [ 1788.509588] Key type .llcrypt unregistered [ 1788.516532] Key type ._llcrypt unregistered [ 1836.042499] Key type ._llcrypt registered [ 1836.046035] Key type .llcrypt registered [ 1836.184102] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 1837.480818] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1837.739535] alg: No test for adler32 (adler32-zlib) [ 1838.834929] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1839.220373] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 1840.903132] Key type lgssc registered [ 1842.382908] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1850.372504] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1850.407963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1851.839345] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1851.857122] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1852.151678] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1857.039658] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 1866.949859] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1867.054512] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1867.110365] Lustre: 57598:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1867.263066] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1867.430963] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1871.797232] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 1880.075610] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1886.348553] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1892.778292] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1893.032684] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1893.241188] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1899.047731] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 1902.224298] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1906.861599] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1913.653570] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-03 9h33m59s [ 1915.319585] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1924.298224] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1927.488229] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1927.591267] LustreError: 60223:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1948.163298] Lustre: lustre-MDT0000: Client afdce17d-ecf8-4afa-88ba-d80a997cccb6 (at 192.168.201.13@tcp) reconnecting [ 1967.607136] LustreError: 60223:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1970.286788] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1972.800494] Lustre: DEBUG MARKER: done [ 1978.849106] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1978.854237] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1978.862267] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1979.375876] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1979.383321] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1980.520395] Lustre: server umount lustre-OST0000 complete [ 1984.480910] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1984.504865] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1984.506843] Lustre: Skipped 1 previous similar message [ 1984.526277] Lustre: Skipped 3 previous similar messages [ 1990.684083] Lustre: server umount lustre-MDT0000 complete [ 1994.354171] LustreError: 57589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442521 with bad export cookie 6501497082057442888 [ 1994.361829] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1994.375874] LustreError: 57589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1994.724247] Lustre: server umount lustre-MDT0001 complete [ 2004.780256] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 2007.522631] Key type lgssc unregistered [ 2007.872495] LNet: 61926:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2007.879383] LNetError: 61926:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2007.896656] LNet: Removed LNI 192.168.201.113@tcp [ 2008.690654] Key type .llcrypt unregistered [ 2008.692770] Key type ._llcrypt unregistered [ 2055.777949] Key type ._llcrypt registered [ 2055.780513] Key type .llcrypt registered [ 2055.917627] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 2057.341095] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2057.430165] alg: No test for adler32 (adler32-zlib) [ 2058.479612] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2058.765477] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 2060.456764] Key type lgssc registered [ 2061.691941] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2064.662482] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2064.678086] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2065.932770] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2065.949453] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2066.142993] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2067.263809] Lustre: Failing over lustre-MDT0000 [ 2067.713079] Lustre: server umount lustre-MDT0000 complete [ 2074.592424] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 09:36:41 (1788442601) [ 2081.136794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2081.449843] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2086.783757] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 2094.987882] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2095.078783] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2095.100191] Lustre: 64727:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2095.277369] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2099.403160] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 2106.998583] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2112.413683] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2119.512795] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2119.708876] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2119.969274] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2126.249642] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 2127.589954] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2133.970155] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2140.551560] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2151.908448] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2151.912974] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2151.924198] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2153.446828] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2153.464302] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2153.500145] Lustre: Skipped 1 previous similar message [ 2157.197747] Lustre: server umount lustre-MDT0000 complete [ 2162.241418] LustreError: 65474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442689 with bad export cookie 1975561437235318508 [ 2162.248721] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2162.252557] LustreError: 65474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2162.836365] Lustre: server umount lustre-MDT0001 complete [ 2167.382486] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2174.167890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2179.039908] Lustre: 63651:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788442690/real 1788442690] req@00000000c10a01d7 x1875317972051328/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788442706 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2179.063864] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2183.136298] Lustre: 63653:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788442694/real 1788442694] req@00000000ee573bb4 x1875317972051456/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788442710 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2183.539881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2184.159616] Lustre: 63652:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788442695/real 1788442695] req@000000009bfed783 x1875317972051584/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788442711 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2187.295898] LustreError: 63649:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000066718f89 x1875317972052352/t0(0) o250->MGC192.168.201.113@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2187.708756] LustreError: 66578:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2187.742061] LustreError: 66578:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2187.850265] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2187.971050] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2188.831252] Lustre: 63650:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788442699/real 1788442699] req@000000005a12f866 x1875317972051840/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788442715 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2188.834656] LustreError: 66579:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2189.861474] LustreError: 68495:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2189.884876] LustreError: 68495:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2193.558575] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 2193.887111] Lustre: 63651:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788442705/real 1788442705] req@000000007ee0ab06 x1875317972051968/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788442721 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2194.920470] LustreError: 68497:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2194.950851] LustreError: 68497:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2200.035495] LustreError: 68495:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2200.058043] LustreError: 68495:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2202.698826] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2203.140703] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2207.897746] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 2208.250336] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2209.323137] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2216.381082] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2222.678341] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2229.729681] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2229.734350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2229.744082] Lustre: Skipped 1 previous similar message [ 2229.767955] Lustre: Skipped 4 previous similar messages [ 2234.854790] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2234.863763] Lustre: Skipped 2 previous similar messages [ 2235.359505] Lustre: server umount lustre-MDT0000 complete [ 2239.442567] LustreError: 64720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442766 with bad export cookie 1975561437235320783 [ 2239.445078] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2239.456336] LustreError: 64720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2239.660223] Lustre: server umount lustre-MDT0001 complete [ 2247.275555] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2256.351212] Lustre: 63651:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788442767/real 1788442767] req@00000000b863c17d x1875317972080640/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788442783 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2256.400993] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2256.417243] Lustre: Skipped 1 previous similar message [ 2259.821876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2264.616875] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x1b6a9aa9025f1d32 [ 2264.660120] Lustre: MGC192.168.201.113@tcp: Connection restored to 0@lo (at 0@lo) [ 2265.282880] LustreError: 66578:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2265.546298] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2265.723605] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2272.336471] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 2281.954093] LustreError: 71180:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2281.985922] LustreError: 71180:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 2282.757560] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2285.549653] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2288.943630] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 2299.406993] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2306.293224] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2314.208297] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2314.218657] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2314.243159] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2320.720628] Lustre: server umount lustre-MDT0000 complete [ 2321.381658] LustreError: 71182:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2321.421476] LustreError: 71182:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2326.193204] LustreError: 64719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442853 with bad export cookie 1975561437235322162 [ 2326.197797] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2326.201972] LustreError: 64719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2326.841412] Lustre: server umount lustre-MDT0001 complete [ 2334.604951] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2338.244513] Lustre: DEBUG MARKER: files compared the same [ 2340.242209] Lustre: server umount lustre-OST0000 complete [ 2358.204682] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 2360.546639] Key type lgssc unregistered [ 2360.772831] LNet: 74462:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2360.780750] LNetError: 74462:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2360.798254] LNet: Removed LNI 192.168.201.113@tcp [ 2361.634601] Key type .llcrypt unregistered [ 2361.638497] Key type ._llcrypt unregistered [ 2386.892253] Key type ._llcrypt registered [ 2386.894798] Key type .llcrypt registered [ 2386.993513] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 2388.650495] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2388.670377] alg: No test for adler32 (adler32-zlib) [ 2389.764600] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2390.110633] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 2391.880093] Key type lgssc registered [ 2393.574897] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2401.334362] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2401.356553] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2402.784627] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2407.572587] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug +malloc all [ 2412.011040] LustreError: 75881:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2412.036286] LustreError: 75881:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2417.120036] LustreError: 75882:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2417.188956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2417.507710] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2422.741603] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug +malloc all [ 2431.139385] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2438.549491] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2447.102825] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2447.453436] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2448.513321] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2456.255457] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug +malloc all [ 2466.792670] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2478.561416] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2478.565324] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2478.571564] Lustre: Skipped 1 previous similar message [ 2478.571980] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2478.578189] Lustre: Skipped 1 previous similar message [ 2483.693541] LustreError: 77705:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2483.721943] LustreError: 77705:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2483.844787] Lustre: server umount lustre-OST0000 complete [ 2489.312774] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2489.327608] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2489.361596] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2493.920273] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2493.939571] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2495.565778] Lustre: server umount lustre-MDT0000 complete [ 2499.041795] LustreError: 75882:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2499.084274] LustreError: 75882:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2500.917434] LustreError: 75864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443028 with bad export cookie 550562273753113446 [ 2500.934868] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2501.304129] Lustre: server umount lustre-MDT0001 complete [ 2515.860444] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 2519.277581] Key type lgssc unregistered [ 2519.574811] LNet: 79421:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2519.582561] LNetError: 79421:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2519.597063] LNet: Removed LNI 192.168.201.113@tcp [ 2520.465165] Key type .llcrypt unregistered [ 2520.475879] Key type ._llcrypt unregistered [ 2547.595484] Key type ._llcrypt registered [ 2547.599042] Key type .llcrypt registered [ 2547.724778] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 2549.020324] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2549.134476] alg: No test for adler32 (adler32-zlib) [ 2550.363171] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2550.609381] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 2552.351993] Key type lgssc registered [ 2553.698684] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2561.988036] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2562.006238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2577.567677] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2582.687349] LustreError: 80814:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.113@tcp: failed processing log, type 4: rc = -110 [ 2609.375747] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2616.469822] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 2626.588677] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2635.140657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2635.261071] Lustre: *** cfs_fail_loc=706, val=0*** [ 2635.662839] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2635.744344] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2640.878941] LustreError: 80840:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2640.907947] LustreError: 80840:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2641.078591] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 2643.952246] LustreError: 80839:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2643.980664] LustreError: 80839:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2649.064196] LustreError: 81897:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2649.102431] LustreError: 81897:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2651.006306] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2651.405777] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2656.487190] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 2665.102571] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2672.177622] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2676.709615] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2676.716792] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2676.732406] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2677.222152] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2677.244094] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2677.260297] Lustre: Skipped 1 previous similar message [ 2681.664359] Lustre: server umount lustre-OST0000 complete [ 2687.461711] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2687.480598] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2687.492935] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2692.583991] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2692.589147] Lustre: Skipped 2 previous similar messages [ 2693.477707] Lustre: server umount lustre-MDT0000 complete [ 2697.698118] LustreError: 82897:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2697.740965] LustreError: 82897:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2698.460798] LustreError: 80821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443225 with bad export cookie 16240343923045498835 [ 2698.484858] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2698.487888] LustreError: 80821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2699.048579] Lustre: server umount lustre-MDT0001 complete [ 2713.469496] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 2716.440512] Key type lgssc unregistered [ 2716.727828] LNet: 84437:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2716.738292] LNetError: 84437:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2717.801550] LNet: Removed LNI 192.168.201.113@tcp [ 2718.677382] Key type .llcrypt unregistered [ 2718.679940] Key type ._llcrypt unregistered [ 2744.547343] Key type ._llcrypt registered [ 2744.548975] Key type .llcrypt registered [ 2744.629581] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 2745.576163] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2745.610984] alg: No test for adler32 (adler32-zlib) [ 2746.754619] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2746.963973] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 2748.583235] Key type lgssc registered [ 2749.543080] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2757.670187] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2757.685579] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2763.356372] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 2772.106179] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2772.706565] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2777.318441] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 2778.090343] LustreError: 86543:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2778.119626] LustreError: 86543:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2782.184674] LustreError: 86547:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2782.210163] LustreError: 86547:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2786.291934] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2786.617689] LustreError: 86543:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2786.649421] LustreError: 86543:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2786.669587] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2791.917347] LustreError: 87289:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2791.933986] LustreError: 87289:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2792.750157] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 2797.027037] LustreError: 86546:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2797.045590] LustreError: 86546:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2802.031641] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2802.042146] Lustre: MGS: Client 5838b93d-3975-4ae9-b821-09b0f9deab80 (at 0@lo) reconnecting [ 2802.075437] LustreError: 86557:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2802.147355] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [ 2802.299628] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2802.403067] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2807.299780] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 2823.145544] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2823.156716] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2823.163950] Lustre: Skipped 1 previous similar message [ 2823.176984] Lustre: Skipped 3 previous similar messages [ 2828.287218] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2828.306476] Lustre: Skipped 1 previous similar message [ 2829.302469] Lustre: server umount lustre-OST0000 complete [ 2833.376815] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2833.377128] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2833.381292] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2833.388302] Lustre: Skipped 1 previous similar message [ 2838.497721] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2838.505012] Lustre: Skipped 2 previous similar messages [ 2839.186167] Lustre: server umount lustre-MDT0000 complete [ 2843.904496] Lustre: server umount lustre-MDT0001 complete [ 2847.819458] Lustre: server umount MGS complete [ 2862.483383] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 09:49:48 (1788443388) [ 2907.609781] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [ 2920.526825] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 2961.122731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2966.675665] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 2976.944825] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2977.160787] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2977.214181] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2977.306614] Lustre: lustre-MDT0001: new disk, initializing [ 2977.391140] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2977.456613] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2977.467597] Lustre: 93735:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2977.486230] LustreError: 93735:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2982.176260] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 2994.859192] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2995.197541] Lustre: lustre-OST0000: new disk, initializing [ 2995.205814] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2995.214325] Lustre: 94503:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2995.337034] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3001.773582] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 3006.058117] Lustre: 94972:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@000000002611a29d x1875318694518272/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788443549 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3012.934789] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3020.889661] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3020.898954] Lustre: MGS: Client bedc848c-b208-4962-b9e7-9f11a1ee9466 (at 0@lo) reconnecting [ 3020.947407] Lustre: 95548:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3020.973226] LustreError: 93721:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 3020.986475] LustreError: 93721:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 3021.019394] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3021.132785] Lustre: lustre-MDT0000: new disk, initializing [ 3021.333951] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3021.363864] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3021.381133] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3022.444389] Lustre: 95572:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@000000002912d83e x1875318694525056/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1788443565 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3027.523241] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3027.539630] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3027.624567] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 3027.663259] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 3058.158585] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3058.165859] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3058.168462] Lustre: Skipped 1 previous similar message [ 3058.182689] Lustre: Skipped 1 previous similar message [ 3060.859218] Lustre: server umount lustre-OST0000 complete [ 3068.392706] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3068.409865] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3071.888798] Lustre: server umount lustre-MDT0000 complete [ 3073.510638] LustreError: 93711:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3073.540702] LustreError: 93711:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3082.794710] Lustre: server umount lustre-MDT0001 complete [ 3086.988666] Lustre: server umount MGS complete [ 3101.787659] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 09:53:47 (1788443627) [ 3123.654115] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 3126.303211] Key type lgssc unregistered [ 3126.627644] LNet: 98073:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3126.636286] LNetError: 98073:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3126.656195] LNet: Removed LNI 192.168.201.113@tcp [ 3127.432037] Key type .llcrypt unregistered [ 3127.434225] Key type ._llcrypt unregistered [ 3147.663618] Key type ._llcrypt registered [ 3147.671111] Key type .llcrypt registered [ 3147.805760] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 3149.104606] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3149.134785] alg: No test for adler32 (adler32-zlib) [ 3150.287431] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 3150.700261] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 3152.463222] Key type lgssc registered [ 3153.944540] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3162.819222] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3162.836137] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3164.346830] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3169.286310] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 3173.857711] LustreError: 99297:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3173.908090] LustreError: 99297:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3178.976603] LustreError: 99296:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3181.455091] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3182.109431] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3188.038765] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 3197.033725] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3204.130876] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3211.967884] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3212.212497] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3218.805652] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 3227.136761] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3234.745168] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3248.097142] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3248.110707] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3248.127282] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3250.146269] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3250.155962] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3250.168270] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3250.230102] Lustre: Skipped 1 previous similar message [ 3252.322286] Lustre: server umount lustre-OST0000 complete [ 3258.858799] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3258.866320] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3258.891790] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3262.377495] Lustre: server umount lustre-MDT0000 complete [ 3265.518543] LustreError: 99292:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3265.559579] LustreError: 99292:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3266.300233] LustreError: 99279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443793 with bad export cookie 16172659121303872949 [ 3266.302284] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3266.308449] LustreError: 99279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3266.768929] Lustre: server umount lustre-MDT0001 complete [ 3278.068399] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 3280.873325] Key type lgssc unregistered [ 3281.173517] LNet: 102859:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3281.178447] LNetError: 102859:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3281.192963] LNet: Removed LNI 192.168.201.113@tcp [ 3281.962130] Key type .llcrypt unregistered [ 3281.965432] Key type ._llcrypt unregistered [ 3299.124284] Key type ._llcrypt registered [ 3299.126839] Key type .llcrypt registered [ 3299.301037] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 3300.112750] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3300.189834] alg: No test for adler32 (adler32-zlib) [ 3301.310338] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 3301.562066] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 3303.210796] Key type lgssc registered [ 3304.548526] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3310.245967] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3310.246359] LustreError: 103942:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3310.256394] Lustre: Skipped 1 previous similar message [ 3310.262215] LustreError: 103939:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3310.266985] LustreError: 103942:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4994 [ 3310.275563] LustreError: 103942:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3310.282826] LustreError: 103942:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3310.286518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3311.734662] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3319.642161] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3319.949572] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3323.473803] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 3328.177368] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing lsmod [ 3330.790373] LustreError: 105144:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3330.822556] LustreError: 105148:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3330.827648] LustreError: 105144:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4970 [ 3330.830820] LustreError: 105144:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3330.837781] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3330.843823] LustreError: 105144:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3331.216239] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3335.652450] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3335.662223] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3335.681244] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3336.682446] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3336.689639] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3336.699072] Lustre: Skipped 1 previous similar message [ 3336.705482] Lustre: Skipped 1 previous similar message [ 3340.173052] Lustre: server umount lustre-MDT0000 complete [ 3341.799570] LustreError: 103974:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3341.818440] LustreError: 103974:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 3344.190937] LustreError: 103954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443871 with bad export cookie 10405631141719397912 [ 3344.203660] LustreError: 103954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3344.205131] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3344.225107] Lustre: 103972:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.201.113@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 3344.239956] LustreError: 103975:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3344.241295] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3344.243177] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3344.243185] Lustre: Skipped 1 previous similar message [ 3344.726499] Lustre: server umount lustre-MDT0001 complete [ 3348.456381] Lustre: server umount lustre-OST0000 complete [ 3356.433479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3356.889564] LustreError: 106211:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3357.009075] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3360.891485] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 3362.275023] LustreError: 106212:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3367.395397] LustreError: 106211:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3368.660581] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3369.004294] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3372.891594] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 3380.675142] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3387.462704] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3394.676469] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3395.035740] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3401.226956] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 3402.242228] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3408.675939] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3420.221780] Lustre: server umount lustre-OST0000 complete [ 3429.858322] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3429.880331] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3429.891937] Lustre: Skipped 1 previous similar message [ 3434.976979] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3434.992412] Lustre: Skipped 1 previous similar message [ 3436.279907] Lustre: server umount lustre-MDT0000 complete [ 3440.100078] LustreError: 106212:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3440.105837] Lustre: 103346:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788443951/real 1788443951] req@00000000f87f39f6 x1875319275437056/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788443967 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3440.150795] LustreError: 106212:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3440.249421] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3440.265871] Lustre: Skipped 1 previous similar message [ 3441.454648] LustreError: 106192:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443968 with bad export cookie 10405631141719398598 [ 3441.462934] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3441.467357] LustreError: 106192:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3441.836900] Lustre: server umount lustre-MDT0001 complete [ 3456.440453] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 3460.397988] Key type lgssc unregistered [ 3460.723035] LNet: 109749:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3460.731116] LNetError: 109749:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3461.812702] LNet: Removed LNI 192.168.201.113@tcp [ 3463.049612] Key type .llcrypt unregistered [ 3463.052619] Key type ._llcrypt unregistered [ 3497.741756] Key type ._llcrypt registered [ 3497.746746] Key type .llcrypt registered [ 3497.922716] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 3499.226020] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3499.332951] alg: No test for adler32 (adler32-zlib) [ 3500.440463] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 3500.683753] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 3502.351223] Key type lgssc registered [ 3503.564264] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3512.106803] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3512.129455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3513.586445] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3519.761348] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 3523.041736] LustreError: 111168:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3523.058534] LustreError: 111168:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3528.160684] LustreError: 111169:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3531.859718] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3532.518206] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3538.799167] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 3548.644970] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3556.524593] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3565.903777] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3566.120739] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3574.843288] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 3583.587663] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3584.570638] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3594.264966] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3598.654601] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3602.912990] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3602.931936] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3602.959899] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3604.816892] Lustre: server umount lustre-OST0000 complete [ 3610.080476] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3610.084674] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3610.120109] Lustre: Skipped 1 previous similar message [ 3615.202523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3615.217486] Lustre: Skipped 2 previous similar messages [ 3620.325327] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3620.436793] Lustre: server umount lustre-MDT0000 complete [ 3624.333637] LustreError: 111149:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444151 with bad export cookie 16232304548490683312 [ 3624.338905] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3624.340718] LustreError: 111149:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3624.568109] Lustre: server umount lustre-MDT0001 complete [ 3635.897287] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 3638.810676] Key type lgssc unregistered [ 3639.124567] LNet: 114824:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3639.129037] LNetError: 114824:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3639.145768] LNet: Removed LNI 192.168.201.113@tcp [ 3639.969622] Key type .llcrypt unregistered [ 3639.971809] Key type ._llcrypt unregistered [ 3660.349335] Key type ._llcrypt registered [ 3660.352218] Key type .llcrypt registered [ 3660.486522] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 3661.473010] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3661.702922] alg: No test for adler32 (adler32-zlib) [ 3663.045834] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 3663.373994] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 3665.136376] Key type lgssc registered [ 3666.782622] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3676.103274] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3676.150679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3677.615947] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3682.261411] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 3686.893550] LustreError: 116048:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3686.918104] LustreError: 116048:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3692.006035] LustreError: 116049:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3692.739900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3693.072442] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3697.927134] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 3707.396683] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3715.115726] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3723.970477] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3724.529953] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3731.366527] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 3739.221885] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3747.082417] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3757.572107] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3757.579354] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3757.599913] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3757.607890] Lustre: Skipped 1 previous similar message [ 3762.726178] Lustre: server umount lustre-OST0000 complete [ 3767.780821] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3767.796504] Lustre: Skipped 1 previous similar message [ 3767.809928] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3772.899328] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3772.903912] Lustre: Skipped 2 previous similar messages [ 3773.962223] Lustre: server umount lustre-MDT0000 complete [ 3776.993994] LustreError: 116043:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3777.004152] LustreError: 116043:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3779.275095] LustreError: 116028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444306 with bad export cookie 8919510966347938355 [ 3779.298167] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3780.663628] Lustre: server umount lustre-MDT0001 complete [ 3795.679902] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 3799.690516] Key type lgssc unregistered [ 3799.970314] LNet: 119618:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3799.983580] LNetError: 119618:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3799.998953] LNet: Removed LNI 192.168.201.113@tcp [ 3800.912684] Key type .llcrypt unregistered [ 3800.915278] Key type ._llcrypt unregistered [ 3827.660521] Key type ._llcrypt registered [ 3827.662743] Key type .llcrypt registered [ 3827.778860] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 3828.944027] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3828.994444] alg: No test for adler32 (adler32-zlib) [ 3830.476740] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 3830.751323] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 3832.415561] Key type lgssc registered [ 3833.949886] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3843.653865] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3843.680452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3845.198951] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3851.896692] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 3855.330135] LustreError: 121047:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3855.347542] LustreError: 121047:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3860.459769] LustreError: 121048:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3860.906335] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3861.467331] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3866.402341] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 3874.161161] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3880.556694] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3887.647705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3887.949990] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3890.049961] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3895.051967] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 3904.752325] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3911.913574] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3916.613780] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3916.621285] Lustre: Skipped 2 previous similar messages [ 3921.006408] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3921.071274] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3921.248311] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3921.253610] Lustre: Skipped 2 previous similar messages [ 3931.805494] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3931.821388] Lustre: Skipped 1 previous similar message [ 3937.690684] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.201.13@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3937.695933] Lustre: Skipped 2 previous similar messages [ 3945.079832] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.201.13@tcp 192.168.0.[2,10]@tcp [ 3952.041165] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.201.13/24@tcp in log lustre-MDT0000 [ 3952.049375] Lustre: Skipped 2 previous similar messages [ 3959.869208] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.13/24@tcp [ 3959.872347] Lustre: Skipped 1 previous similar message [ 3966.946112] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3966.962892] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3966.968505] LustreError: Skipped 2 previous similar messages [ 3967.000913] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3967.983173] Lustre: server umount lustre-OST0000 complete [ 3973.107386] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3973.118471] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3973.127900] Lustre: Skipped 1 previous similar message [ 3978.213383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3978.228339] Lustre: Skipped 2 previous similar messages [ 3983.330845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3983.339420] Lustre: Skipped 1 previous similar message [ 3984.351572] Lustre: 120361:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788444495/real 1788444495] req@00000000628de884 x1875319830155904/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788444511 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3984.387980] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3986.399935] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3986.635679] Lustre: server umount lustre-MDT0000 complete [ 3989.475535] LustreError: 121047:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3989.479771] Lustre: 120359:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788444500/real 1788444500] req@000000007a487d67 x1875319830158080/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788444516 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3989.501517] LustreError: 121047:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3990.491654] LustreError: 124083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444517 with bad export cookie 3520645896353845428 [ 3990.499130] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3990.501429] LustreError: 124083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3991.002916] Lustre: server umount lustre-MDT0001 complete [ 4002.373837] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 4005.727466] Key type lgssc unregistered [ 4006.119550] LNet: 126236:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4006.131364] LNetError: 126236:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4006.162488] LNet: Removed LNI 192.168.201.113@tcp [ 4007.419410] Key type .llcrypt unregistered [ 4007.426467] Key type ._llcrypt unregistered [ 4041.774861] Key type ._llcrypt registered [ 4041.778516] Key type .llcrypt registered [ 4041.896214] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 4043.496774] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4043.596104] alg: No test for adler32 (adler32-zlib) [ 4044.774526] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 4045.076756] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 4046.830911] Key type lgssc registered [ 4048.148946] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4056.689613] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4056.726526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4058.290133] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 4058.312980] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.201.13/24@tcp [ 4058.315727] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4062.989337] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 4068.840895] LustreError: 127799:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4068.870684] LustreError: 127799:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4072.742889] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4073.150812] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4073.195632] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.13/24@tcp [ 4073.263810] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4078.688045] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 4089.895192] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4096.887402] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4105.070592] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4105.475144] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4112.924823] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 4120.657305] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4128.024126] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 4143.587025] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4143.587600] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4143.625848] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4143.657618] Lustre: Skipped 1 previous similar message [ 4148.611631] Lustre: server umount lustre-OST0000 complete [ 4153.824894] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4153.831947] Lustre: Skipped 1 previous similar message [ 4153.834890] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4158.960850] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4158.973877] Lustre: Skipped 2 previous similar messages [ 4160.218725] Lustre: server umount lustre-MDT0000 complete [ 4163.042985] LustreError: 129895:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4163.060469] LustreError: 129895:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4164.514829] LustreError: 127781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444691 with bad export cookie 16957729466868837905 [ 4164.528158] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4164.789130] Lustre: server umount lustre-MDT0001 complete [ 4178.047746] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 4181.829886] Key type lgssc unregistered [ 4182.263071] LNet: 131416:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4182.278510] LNetError: 131416:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4182.326166] LNet: Removed LNI 192.168.201.113@tcp [ 4183.793775] Key type .llcrypt unregistered [ 4183.803809] Key type ._llcrypt unregistered [ 4222.258767] Key type ._llcrypt registered [ 4222.260434] Key type .llcrypt registered [ 4222.372120] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [ 4239.011654] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 4240.536711] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4240.762652] alg: No test for adler32 (adler32-zlib) [ 4241.896876] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 4242.070373] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 4243.697057] Key type lgssc registered [ 4245.196475] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4285.998868] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4286.035545] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4287.426420] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4287.481741] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4287.553787] Lustre: lustre-MDT0000: new disk, initializing [ 4287.675653] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4287.719271] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4293.945673] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 4305.636745] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4305.747684] Lustre: 134820:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4305.802166] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4305.812194] Lustre: Skipped 1 previous similar message [ 4305.895438] Lustre: lustre-MDT0001: new disk, initializing [ 4305.970725] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4306.010243] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4306.019852] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4310.218384] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 4319.550585] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4325.452859] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4334.197261] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4334.718084] Lustre: lustre-OST0000: new disk, initializing [ 4334.725701] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4334.736215] Lustre: 136790:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4334.865690] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4342.414521] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4342.431888] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4342.596800] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4343.559682] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 4354.496640] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4360.728707] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4361.078381] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4366.778508] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4367.136628] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4371.427232] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4371.431599] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4371.435407] Lustre: Skipped 1 previous similar message [ 4375.441505] Lustre: server umount lustre-OST0000 complete [ 4381.664238] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4381.667582] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4381.677708] Lustre: Skipped 1 previous similar message [ 4381.681201] Lustre: Skipped 1 previous similar message [ 4386.786904] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4386.798322] Lustre: Skipped 2 previous similar messages [ 4389.144295] Lustre: server umount lustre-MDT0000 complete [ 4393.270192] LustreError: 136789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444920 with bad export cookie 13821136234201752217 [ 4393.272421] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4393.281166] LustreError: 136789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4393.588205] Lustre: server umount lustre-MDT0001 complete [ 4403.742984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4404.543983] LustreError: 139008:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4404.607905] LustreError: 139008:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4404.671037] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4409.839501] LustreError: 139009:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4409.889786] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 4413.927029] LustreError: 139008:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4419.052878] LustreError: 139009:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4420.778345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4421.243183] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4426.015683] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 4436.217794] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4443.434412] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4451.259853] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4451.757189] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4459.325440] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 4468.194738] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4473.640575] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4474.960746] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 4480.450772] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4480.711378] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4492.250541] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4492.451830] Lustre: lustre-OST0001: new disk, initializing [ 4492.454271] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4492.458301] Lustre: 142389:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4492.534721] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4499.623995] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4499.645217] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4499.803550] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4499.875145] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 4510.072467] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4510.348679] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4516.211643] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4516.481622] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4524.668490] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff88e2c73b6000.ost_server_uuid 50 [ 4526.494518] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff88e2c73b6000.ost_server_uuid in IDLE state after 0 sec [ 4540.896758] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4540.905156] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4540.916414] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4540.919390] Lustre: Skipped 1 previous similar message [ 4540.931711] Lustre: Skipped 1 previous similar message [ 4546.026417] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4546.031981] Lustre: Skipped 1 previous similar message [ 4554.207578] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4554.279837] Lustre: server umount lustre-OST0001 complete [ 4556.262504] LustreError: 142741:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4556.281773] LustreError: 142741:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4558.308440] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4558.321215] LustreError: Skipped 1 previous similar message [ 4558.330500] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4558.359524] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4558.369033] Lustre: Skipped 2 previous similar messages [ 4561.391668] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4564.471127] Lustre: server umount lustre-OST0000 complete [ 4569.573124] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4569.588714] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4569.601590] Lustre: Skipped 1 previous similar message [ 4576.737888] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4576.745786] Lustre: Skipped 6 previous similar messages [ 4582.880181] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4583.427673] Lustre: server umount lustre-MDT0000 complete [ 4585.954910] LustreError: 139008:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4585.975448] LustreError: 139008:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4588.169728] LustreError: 138989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445115 with bad export cookie 13821136234201753099 [ 4588.176773] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4588.183620] LustreError: 138989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4588.549210] Lustre: server umount lustre-MDT0001 complete [ 4609.397132] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 4612.060167] Key type lgssc unregistered [ 4612.419270] LNet: 145417:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4612.434272] LNetError: 145417:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4612.469652] LNet: Removed LNI 192.168.201.113@tcp [ 4613.259379] Key type .llcrypt unregistered [ 4613.261798] Key type ._llcrypt unregistered [ 4659.305635] Key type ._llcrypt registered [ 4659.307693] Key type .llcrypt registered [ 4659.387777] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [ 4676.436071] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 4678.437632] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4678.497689] alg: No test for adler32 (adler32-zlib) [ 4679.706811] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 4680.001471] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 4681.703359] Key type lgssc registered [ 4683.080327] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4724.999663] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4725.021811] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4726.280240] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4726.299715] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4726.363129] Lustre: lustre-MDT0000: new disk, initializing [ 4726.424734] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4726.443343] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4731.976658] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 4743.445153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4743.660672] Lustre: 149577:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4743.773956] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4743.786164] Lustre: Skipped 1 previous similar message [ 4743.895341] Lustre: lustre-MDT0001: new disk, initializing [ 4744.005140] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4744.068744] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4744.080238] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4748.685748] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 4761.476789] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4772.184282] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4782.288120] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4782.527479] Lustre: lustre-OST0000: new disk, initializing [ 4782.530937] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4782.534581] Lustre: 151547:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4782.608253] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4790.004512] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 4791.389104] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4791.407488] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4791.660127] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4800.419710] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4805.162252] Lustre: Failing over lustre-OST0000 [ 4805.356485] Lustre: server umount lustre-OST0000 complete [ 4823.519229] Lustre: 147801:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788445334/real 1788445334] req@00000000ac886964 x1875320721411584/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1788445350 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4823.521224] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4823.565921] Lustre: 147801:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 4823.577440] Lustre: Skipped 1 previous similar message [ 4824.686954] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4825.143965] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4825.162356] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4828.705249] Lustre: 147800:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788445339/real 1788445339] req@0000000078c3e5fc x1875320721412608/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788445355 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4828.751363] Lustre: 147800:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4833.212739] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4833.247840] Lustre: 147798:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788445344/real 1788445344] req@000000000e2f90cf x1875320721413888/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788445360 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4833.277143] Lustre: 147798:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4833.333726] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 4833.342257] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4833.850706] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 4838.373197] Lustre: 147799:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788445349/real 1788445349] req@00000000f7e6fd00 x1875320721414912/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788445365 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4838.431102] Lustre: 147799:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4838.745833] Lustre: Failing over lustre-MDT0000 [ 4839.035425] Lustre: server umount lustre-MDT0000 complete [ 4841.444665] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4841.462673] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4843.495682] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4843.518891] Lustre: Skipped 1 previous similar message [ 4848.562899] LustreError: 149585:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4848.588503] LustreError: 149585:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 4853.678817] LustreError: 149585:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4853.697966] LustreError: 149585:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4858.800297] LustreError: 149584:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4858.824830] LustreError: 149584:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4859.727139] Lustre: 147799:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788445370/real 1788445370] req@00000000be4ef49e x1875320721426432/t0(0) o400->MGC192.168.201.113@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788445386 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4859.754094] Lustre: 147799:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4859.762963] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4860.687214] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4864.942358] LustreError: 149584:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4864.962629] LustreError: 149584:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 4869.043291] LustreError: 149585:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4869.064923] LustreError: 149585:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4869.589366] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4869.676517] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4874.175133] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4874.740547] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4874.761742] Lustre: Skipped 1 previous similar message [ 4874.803789] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4874.862555] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4876.145853] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 4885.472399] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4885.477370] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4885.491697] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4890.086719] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4890.090889] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4890.094493] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4890.117314] Lustre: Skipped 1 previous similar message [ 4891.731764] Lustre: server umount lustre-OST0000 complete [ 4900.324603] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4900.324989] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4900.328626] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4900.365214] Lustre: Skipped 1 previous similar message [ 4900.393704] Lustre: Skipped 1 previous similar message [ 4902.184380] Lustre: server umount lustre-MDT0000 complete [ 4905.457156] LustreError: 149588:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4905.494274] LustreError: 149588:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4906.179303] LustreError: 149568:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445433 with bad export cookie 8491308120160942616 [ 4906.202288] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4906.694207] Lustre: server umount lustre-MDT0001 complete [ 4917.700993] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 4921.817060] Key type lgssc unregistered [ 4922.074753] LNet: 155210:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4922.084106] LNetError: 155210:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4922.106952] LNet: Removed LNI 192.168.201.113@tcp [ 4923.310840] Key type .llcrypt unregistered [ 4923.313350] Key type ._llcrypt unregistered [ 4962.449308] Key type ._llcrypt registered [ 4962.451502] Key type .llcrypt registered [ 4962.557948] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 4963.842155] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4963.892094] alg: No test for adler32 (adler32-zlib) [ 4964.990406] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 4965.234539] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 4966.919956] Key type lgssc registered [ 4968.374776] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4976.301902] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4976.318385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4977.751371] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4981.802791] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 4987.875411] LustreError: 156737:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4987.893532] LustreError: 156737:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4991.535582] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4992.993186] LustreError: 156738:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4996.059513] LustreError: 156766:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4996.078215] LustreError: 156766:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4996.087715] LustreError: 156766:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 0, failed: rc = -5 [ 4996.439315] Lustre: server umount lustre-MDT0000 complete [ 5012.204536] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [ 5022.956466] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 5071.966600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5072.342467] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5072.427635] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5072.608977] Lustre: lustre-MDT0000: new disk, initializing [ 5072.726207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5072.791518] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5079.899124] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 5091.867149] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5092.017737] Lustre: 160070:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5092.105589] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5092.109604] Lustre: Skipped 1 previous similar message [ 5092.176531] Lustre: lustre-MDT0001: new disk, initializing [ 5092.312867] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5092.382160] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5092.393949] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5097.412797] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 5107.532446] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5113.539783] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5121.308524] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5121.789930] Lustre: lustre-OST0000: new disk, initializing [ 5121.793796] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5121.807708] Lustre: 162039:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5121.984981] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5128.896207] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 5131.356446] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5131.366456] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5131.459241] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5138.306504] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5143.557103] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 5145.416898] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 10:27:51 (1788445671) [ 5408.229200] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5408.232940] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5408.254395] Lustre: Skipped 1 previous similar message [ 5413.344244] LustreError: 160083:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5413.368109] LustreError: 160083:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5413.415409] Lustre: server umount lustre-MDT0000 complete [ 5417.195347] LustreError: 160062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445944 with bad export cookie 1004899022862145121 [ 5417.196627] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5417.203504] LustreError: 160062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5418.463964] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5418.465107] LustreError: 160078:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5418.480233] Lustre: Skipped 1 previous similar message [ 5418.481792] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5418.481799] Lustre: Skipped 2 previous similar messages [ 5422.496847] LustreError: 162710:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5422.498109] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5423.743078] Lustre: server umount lustre-MDT0001 complete [ 5434.088861] Lustre: server umount lustre-OST0000 complete [ 5444.962684] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [ 5456.180096] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 5502.649839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5503.127363] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5503.175287] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5503.244982] Lustre: lustre-MDT0000: new disk, initializing [ 5503.302346] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5503.317994] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5508.373366] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 5522.241482] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5522.404443] Lustre: 166706:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5522.498552] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5522.513959] Lustre: Skipped 1 previous similar message [ 5522.671761] Lustre: lustre-MDT0001: new disk, initializing [ 5522.804621] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5522.879896] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5522.890909] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5528.437552] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 5542.113761] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5551.147984] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5559.643840] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5560.202995] Lustre: lustre-OST0000: new disk, initializing [ 5560.217352] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5560.239293] Lustre: 168674:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5560.438963] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5567.052227] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5567.075530] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5567.244584] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5567.736640] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 5578.569620] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5583.509804] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5583.785187] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5588.324664] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5588.475766] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5593.059566] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5593.066414] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5593.066765] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5593.067878] Lustre: Skipped 1 previous similar message [ 5593.094321] Lustre: Skipped 1 previous similar message [ 5596.990899] Lustre: server umount lustre-OST0000 complete [ 5603.299195] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5603.310184] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5603.318134] Lustre: Skipped 1 previous similar message [ 5603.340975] Lustre: Skipped 1 previous similar message [ 5607.393543] Lustre: server umount lustre-MDT0000 complete [ 5608.424657] LustreError: 168599:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5608.439501] LustreError: 168599:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5611.404657] LustreError: 166696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446138 with bad export cookie 1004899022862303958 [ 5611.405946] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5611.418637] LustreError: 166696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5611.898954] Lustre: server umount lustre-MDT0001 complete [ 5621.310308] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 10:35:47 (1788446147) [ 5639.494123] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [ 5650.264391] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 5692.772574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5692.968396] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5692.990383] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5693.058991] Lustre: lustre-MDT0000: new disk, initializing [ 5693.099508] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5693.108440] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5697.825137] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 5708.722639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5708.839692] Lustre: 173429:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5709.120979] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5712.963718] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 5722.564989] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5728.910685] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5735.268329] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5735.690568] Lustre: lustre-OST0000: new disk, initializing [ 5735.697204] Lustre: Skipped 1 previous similar message [ 5735.702462] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5735.708300] Lustre: Skipped 2 previous similar messages [ 5735.716534] Lustre: 175408:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5741.153675] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5741.161382] Lustre: Skipped 1 previous similar message [ 5741.181602] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5741.349071] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5742.465787] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 5751.873858] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5762.017519] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5762.026219] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5762.034298] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5762.046252] Lustre: Skipped 1 previous similar message [ 5762.066970] Lustre: Skipped 1 previous similar message [ 5765.883612] Lustre: server umount lustre-OST0000 complete [ 5770.721977] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5770.740792] LustreError: Skipped 1 previous similar message [ 5770.752702] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5775.724412] Lustre: server umount lustre-MDT0000 complete [ 5777.380992] LustreError: 173441:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5777.405715] LustreError: 173441:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5780.224145] LustreError: 173421:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446307 with bad export cookie 1004899022862304833 [ 5780.226585] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5780.233897] LustreError: 173421:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5780.771321] Lustre: server umount lustre-MDT0001 complete [ 5789.300492] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 10:38:35 (1788446315) [ 5805.635233] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [ 5817.080741] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 5855.331872] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5855.502657] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5855.523922] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5855.589215] Lustre: lustre-MDT0000: new disk, initializing [ 5855.670838] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5855.678968] Lustre: Skipped 2 previous similar messages [ 5855.693708] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5860.042424] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 5869.699548] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5869.809782] Lustre: 179731:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5869.914413] LustreError: 179742:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5870.204874] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5874.599199] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 5883.853522] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5889.767688] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5897.119991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5897.360812] Lustre: 181708:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5903.923697] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5904.048066] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5904.357405] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 5913.726583] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5924.832797] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5924.835799] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5924.849887] LustreError: Skipped 1 previous similar message [ 5924.851210] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5924.851216] Lustre: Skipped 3 previous similar messages [ 5924.907259] Lustre: Skipped 2 previous similar messages [ 5926.532519] Lustre: server umount lustre-OST0000 complete [ 5944.287939] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5944.492673] Lustre: server umount lustre-MDT0000 complete [ 5945.314718] LustreError: 179738:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5945.339515] LustreError: 179738:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5948.258597] LustreError: 179724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446475 with bad export cookie 1004899022862305925 [ 5948.262421] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5948.274353] LustreError: 179724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5948.549399] Lustre: server umount lustre-MDT0001 complete [ 5959.668369] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 5962.305480] Key type lgssc unregistered [ 5962.639485] LNet: 183567:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5962.652517] LNetError: 183567:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5962.673575] LNet: Removed LNI 192.168.201.113@tcp [ 5963.705318] Key type .llcrypt unregistered [ 5963.707579] Key type ._llcrypt unregistered [ 5988.845986] Key type ._llcrypt registered [ 5988.851346] Key type .llcrypt registered [ 5989.002984] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 5990.230031] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5990.286945] alg: No test for adler32 (adler32-zlib) [ 5991.741765] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 5991.995237] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 5993.663176] Key type lgssc registered [ 5994.944780] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6002.857180] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6002.878906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6004.501560] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6008.714877] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6013.926287] LustreError: 184982:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6013.953700] LustreError: 184982:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6017.515347] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6018.000550] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6021.866257] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6028.704450] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6034.429102] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6042.572990] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6042.994019] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6049.147045] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6049.150396] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 6057.488596] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6079.969433] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6079.979378] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6079.983733] LustreError: Skipped 1 previous similar message [ 6079.984871] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6080.013362] Lustre: Skipped 1 previous similar message [ 6083.686356] Lustre: server umount lustre-OST0000 complete [ 6089.702082] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6089.709638] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6089.730025] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6089.733651] Lustre: Skipped 1 previous similar message [ 6093.906580] Lustre: server umount lustre-MDT0000 complete [ 6095.332198] LustreError: 184976:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6095.357646] LustreError: 184976:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6097.933345] LustreError: 184963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446625 with bad export cookie 6622110603569984633 [ 6097.943260] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6098.270249] Lustre: server umount lustre-MDT0001 complete [ 6109.692991] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 6112.386898] Key type lgssc unregistered [ 6112.684667] LNet: 188547:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6112.697078] LNetError: 188547:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6112.724921] LNet: Removed LNI 192.168.201.113@tcp [ 6113.853977] Key type .llcrypt unregistered [ 6113.856878] Key type ._llcrypt unregistered [ 6141.458352] Key type ._llcrypt registered [ 6141.460486] Key type .llcrypt registered [ 6141.575419] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 6143.783403] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6143.819018] alg: No test for adler32 (adler32-zlib) [ 6144.953745] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 6145.309651] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 6147.039181] Key type lgssc registered [ 6148.110584] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6155.620099] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6155.641817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6156.969222] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6161.982653] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6167.532960] LustreError: 189967:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6167.571637] LustreError: 189967:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6171.268279] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6171.626076] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6176.122202] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6184.071562] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6190.259825] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6196.565297] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6197.342513] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6202.409384] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6203.704642] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6211.153822] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6221.794689] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6221.795090] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6221.806434] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6221.829272] Lustre: Skipped 1 previous similar message [ 6223.290827] Lustre: lustre-OST0000: Not available for connect from 192.168.201.13@tcp (stopping) [ 6223.307263] Lustre: Skipped 1 previous similar message [ 6226.394878] Lustre: server umount lustre-OST0000 complete [ 6234.862563] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff88e2c28c6000.ost_server_uuid 50 [ 6236.792167] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6252.512627] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6252.515653] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6252.539160] Lustre: Skipped 1 previous similar message [ 6256.703225] Lustre: server umount lustre-MDT0000 complete [ 6257.649770] LustreError: 189967:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6257.667707] LustreError: 189967:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6260.685515] LustreError: 189948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446787 with bad export cookie 4228169456909375221 [ 6260.707780] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6260.713981] LustreError: 189948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6260.984611] Lustre: server umount lustre-MDT0001 complete [ 6271.129521] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 10:46:37 (1788446797) [ 6277.719175] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6278.271582] LustreError: 193883:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6278.385208] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6282.947659] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6283.747908] LustreError: 193884:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6288.864387] LustreError: 193883:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6291.399712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6291.677951] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6297.181192] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6304.940702] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6311.426388] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6318.096622] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6318.457664] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6324.271756] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6324.674375] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6330.925680] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6337.713771] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6337.856981] Lustre: lustre-OST0001: new disk, initializing [ 6337.860111] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6337.866293] Lustre: 196721:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6337.956969] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6343.764882] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6345.290323] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6345.300992] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6345.369706] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6353.009237] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6355.971599] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6355.983474] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6355.987517] Lustre: Skipped 1 previous similar message [ 6360.547400] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6360.554458] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6360.568738] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6360.579129] Lustre: Skipped 1 previous similar message [ 6361.650426] Lustre: server umount lustre-OST0000 complete [ 6364.087970] LustreError: 195703:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6368.231344] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6368.413681] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6373.481768] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6373.728312] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6381.544249] LustreError: 195701:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6381.564334] LustreError: 195701:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 6391.778168] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6391.782214] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6391.789552] Lustre: Skipped 1 previous similar message [ 6391.792610] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6391.797438] Lustre: Skipped 3 previous similar messages [ 6393.312083] Lustre: server umount lustre-OST0001 complete [ 6402.965385] Lustre: server umount lustre-MDT0000 complete [ 6406.882779] LustreError: 193863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446934 with bad export cookie 4228169456909376292 [ 6406.894384] LustreError: 193863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6406.894610] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6407.206471] Lustre: server umount lustre-MDT0001 complete [ 6430.543225] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 10:49:17 (1788446957) [ 6437.615342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6437.887065] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6437.966109] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6438.171754] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6442.172095] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6448.597254] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6448.663534] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6448.686264] Lustre: 200165:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6452.831383] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6459.460317] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6464.676456] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6471.544994] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6471.698677] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6477.431208] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6480.035994] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6485.532494] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6491.986422] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6492.035097] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6497.807437] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6505.104708] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6509.543688] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6509.559442] Lustre: Skipped 2 previous similar messages [ 6509.562755] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6509.570298] Lustre: Skipped 5 previous similar messages [ 6522.335679] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6522.401501] Lustre: server umount lustre-OST0000 complete [ 6524.903609] LustreError: 202013:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6524.926816] LustreError: 202013:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 6540.271526] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6540.276256] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6540.291799] Lustre: Skipped 1 previous similar message [ 6540.302189] LustreError: Skipped 1 previous similar message [ 6545.350229] Lustre: server umount lustre-OST0001 complete [ 6550.508341] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6550.518840] Lustre: Skipped 10 previous similar messages [ 6563.295182] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6563.744660] Lustre: server umount lustre-MDT0000 complete [ 6568.385104] LustreError: 200156:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447095 with bad export cookie 4228169456909377692 [ 6568.399242] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6568.407910] LustreError: 200156:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6568.891797] Lustre: server umount lustre-MDT0001 complete [ 6595.928106] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 10:52:02 (1788447122) [ 6613.901156] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [ 6623.850718] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 6665.113240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6665.410600] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6665.494700] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6665.583431] Lustre: lustre-MDT0000: new disk, initializing [ 6665.638236] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6665.647073] Lustre: Skipped 3 previous similar messages [ 6665.659469] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6669.876897] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6679.763343] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6679.962263] Lustre: 208263:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6680.011504] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6680.014860] Lustre: Skipped 1 previous similar message [ 6680.105618] Lustre: lustre-MDT0001: new disk, initializing [ 6680.238744] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6680.242226] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6684.525613] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6693.419591] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6700.103690] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6706.930248] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6707.203849] Lustre: lustre-OST0000: new disk, initializing [ 6707.213831] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6707.223597] Lustre: 210231:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6713.550726] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6713.936207] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6713.943264] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6714.083848] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6723.098570] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6727.991678] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6728.306473] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6733.510470] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6733.666463] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6739.939374] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6739.943733] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6739.944698] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6739.944703] Lustre: Skipped 4 previous similar messages [ 6739.950392] Lustre: Skipped 4 previous similar messages [ 6739.960657] LustreError: Skipped 1 previous similar message [ 6750.175143] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6750.390565] Lustre: server umount lustre-OST0000 complete [ 6759.983241] Lustre: server umount lustre-MDT0000 complete [ 6760.423381] LustreError: 208274:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6760.454653] LustreError: 208274:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [ 6764.026273] LustreError: 208256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447291 with bad export cookie 4228169456909379267 [ 6764.029096] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6764.044295] LustreError: 208256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6764.382664] Lustre: server umount lustre-MDT0001 complete [ 6772.877775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6777.165609] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6785.093150] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6789.547934] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6797.044562] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6803.356380] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6809.978737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6810.396326] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6810.406109] Lustre: Skipped 4 previous similar messages [ 6818.250450] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6826.880895] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6832.771300] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6833.048450] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6838.268577] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6838.449771] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6840.293840] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6840.305311] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6840.308357] LustreError: Skipped 1 previous similar message [ 6840.337854] Lustre: Skipped 3 previous similar messages [ 6846.264815] Lustre: server umount lustre-OST0000 complete [ 6852.705540] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6857.457658] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6886.683124] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6893.558794] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6895.210606] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6895.227955] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6895.228111] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6895.251155] Lustre: Skipped 1 previous similar message [ 6900.130202] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6904.768771] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 3 sec [ 6910.381654] Lustre: server umount lustre-OST0000 complete [ 6916.070313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6916.075183] Lustre: Skipped 12 previous similar messages [ 6924.496599] Lustre: server umount lustre-MDT0000 complete [ 6926.313210] LustreError: 213176:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6926.356149] LustreError: 213176:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 6927.336450] Lustre: 189280:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788447438/real 1788447438] req@00000000303427c6 x1875322257795584/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788447454 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6928.650969] LustreError: 212430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447455 with bad export cookie 4228169456909380149 [ 6928.652535] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6928.662076] LustreError: 212430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6928.896472] Lustre: server umount lustre-MDT0001 complete [ 6938.410934] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 10:57:45 (1788447465) [ 6945.276461] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6950.692171] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6960.146469] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6964.960682] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6973.201477] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6978.842824] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6985.367615] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6992.219456] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 7001.514790] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7006.193276] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7006.364297] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7010.926932] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7011.156108] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7017.809099] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7017.976338] Lustre: lustre-OST0001: new disk, initializing [ 7017.979061] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7017.983414] Lustre: 221785:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7023.220844] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 7025.760300] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7025.766329] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7025.885782] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7033.008237] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7038.065135] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7038.351536] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7044.448350] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7044.638735] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7046.631510] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7046.632078] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7046.642563] Lustre: Skipped 3 previous similar messages [ 7046.656054] LustreError: Skipped 1 previous similar message [ 7052.902510] Lustre: server umount lustre-OST0001 complete [ 7060.801677] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7060.970778] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7065.313969] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7065.621992] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7095.825092] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7096.091420] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7096.098741] Lustre: Skipped 5 previous similar messages [ 7101.421478] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7101.439209] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7101.440617] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7101.459658] Lustre: Skipped 1 previous similar message [ 7103.067773] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 7112.848646] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7114.757985] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7121.918790] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7121.932785] LustreError: Skipped 1 previous similar message [ 7123.205615] Lustre: server umount lustre-OST0001 complete [ 7146.543350] Lustre: server umount lustre-OST0000 complete [ 7157.246108] Lustre: server umount lustre-MDT0000 complete [ 7161.534429] LustreError: 220255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447688 with bad export cookie 4228169456909381115 [ 7161.536278] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7161.552096] LustreError: 220255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7168.036768] Lustre: server umount lustre-MDT0001 complete [ 7190.932990] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 11:01:57 (1788447717) [ 7197.144291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7197.402119] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7197.421812] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7203.146865] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 7211.774085] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7211.845442] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7211.865950] Lustre: 227052:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7215.532144] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 7221.942615] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7227.001744] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7233.168879] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7233.334072] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7239.124255] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 7246.389125] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7255.222291] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7255.314769] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7261.365265] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 7269.002573] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7270.639463] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7275.849495] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7276.040672] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7281.579849] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7281.855980] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7288.391501] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff88e2c66cb000.ost_server_uuid 50 [ 7290.415277] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff88e2c66cb000.ost_server_uuid in IDLE state after 0 sec [ 7291.496855] Lustre: Permanently deactivating lustre-OST0001 [ 7291.507588] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7295.739118] Lustre: Permanently reactivating lustre-OST0001 [ 7295.756780] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7295.764279] Lustre: Skipped 2 previous similar messages [ 7301.612398] LustreError: lustre-OST0001-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 7301.621783] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7301.630616] Lustre: Skipped 12 previous similar messages [ 7304.673329] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7304.721598] Lustre: Skipped 8 previous similar messages [ 7306.684936] Lustre: server umount lustre-OST0001 complete [ 7310.322429] LustreError: 228894:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7310.353224] LustreError: 228894:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 39 previous similar messages [ 7317.074369] Lustre: server umount lustre-OST0000 complete [ 7327.445076] Lustre: server umount lustre-MDT0000 complete [ 7331.416719] LustreError: 227042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447858 with bad export cookie 4228169456909382403 [ 7331.430986] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7331.433455] LustreError: 227042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7331.772851] Lustre: server umount lustre-MDT0001 complete [ 7354.954976] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 11:04:41 (1788447881) [ 7362.237469] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7362.410876] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7362.422753] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7362.426334] Lustre: Skipped 2 previous similar messages [ 7367.257755] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 7376.240852] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7376.349544] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7376.392853] Lustre: 233745:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7381.184479] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 7389.171876] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7394.719757] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7403.202312] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7403.449649] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7403.471379] Lustre: Permanently deactivating lustre-OST0000 [ 7403.520277] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7409.742710] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7412.074297] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 7422.119961] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7430.463455] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7430.669267] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7437.442816] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 7440.040513] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7446.000705] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7447.932741] Lustre: Permanently reactivating lustre-OST0000 [ 7447.953475] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7447.957840] Lustre: Skipped 2 previous similar messages [ 7457.717196] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 48 seconds. I think it's dead, and I am evicting it. exp 00000000fba62668, cur 1788447984 deadline 1788447966 last 1788447936 [ 7457.840874] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7457.846082] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7457.852774] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7457.895287] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7465.441343] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7465.449834] LustreError: Skipped 4 previous similar messages [ 7475.167901] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7475.299301] Lustre: server umount lustre-OST0001 complete [ 7486.129934] Lustre: server umount lustre-OST0000 complete [ 7505.375164] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7505.505845] Lustre: server umount lustre-MDT0000 complete [ 7509.431452] LustreError: 234486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448036 with bad export cookie 4228169456909384069 [ 7509.446386] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7509.447531] LustreError: 234486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7509.706849] Lustre: server umount lustre-MDT0001 complete [ 7523.087770] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 7526.068781] Key type lgssc unregistered [ 7526.518609] LNet: 238639:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7526.532354] LNetError: 238639:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7527.628969] LNet: Removed LNI 192.168.201.113@tcp [ 7528.599390] Key type .llcrypt unregistered [ 7528.601518] Key type ._llcrypt unregistered [ 7552.566389] Key type ._llcrypt registered [ 7552.568943] Key type .llcrypt registered [ 7552.744510] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 7553.679354] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7553.774640] alg: No test for adler32 (adler32-zlib) [ 7555.013679] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 7555.265222] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 7556.895205] Key type lgssc registered [ 7557.994151] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7569.185060] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7569.217038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7570.756616] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7575.041378] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 7580.143067] LustreError: 240108:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7580.158592] LustreError: 240108:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7583.420129] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7583.552492] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7583.576360] Lustre: 240094:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7583.585652] Lustre: Permanently deactivating lustre-MDT0001 [ 7583.600669] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7583.818350] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7588.153491] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 7588.932567] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7595.733111] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7600.662671] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7606.914726] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7607.227119] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7608.233369] LustreError: 241949:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7612.810520] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 7613.414654] LustreError: 241950:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7615.470772] LustreError: 242034:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7615.498825] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7615.501555] LustreError: 242034:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7620.261387] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7620.585599] LustreError: 242034:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7620.608729] LustreError: 242034:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7627.024166] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7627.239183] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7632.406506] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7632.410353] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7634.002686] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 7643.176907] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7647.481985] Lustre: Permanently reactivating lustre-MDT0001 [ 7647.493254] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7647.498232] Lustre: Skipped 2 previous similar messages [ 7651.084093] Lustre: lustre-MDT0001: Client 8284da9e-7ea9-45b8-8a20-6b490b2d80f3 (at 192.168.201.13@tcp) reconnecting [ 7656.855895] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7657.627985] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7657.649776] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7657.678684] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7657.691253] LustreError: 243867:0:(llog_osd.c:2268:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 7657.696045] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7657.707063] LustreError: 243867:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 7657.707736] LustreError: 243867:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.113@tcp: cfg command failed: rc = -108 [ 7657.718086] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 7657.718086] [ 7657.718564] LustreError: 240099:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 7658.437802] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7664.182754] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7664.446571] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7669.951060] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7670.190722] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7672.618812] Lustre: Permanently deactivating lustre-MDT0001 [ 7672.661264] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7672.672085] Lustre: Skipped 2 previous similar messages [ 7679.628831] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7686.113046] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7686.121220] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7686.136128] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7688.676094] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7688.678285] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7688.721048] Lustre: Skipped 2 previous similar messages [ 7692.387984] Lustre: server umount lustre-MDT0000 complete [ 7693.797442] LustreError: 240852:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7693.817326] LustreError: 240852:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 7695.955234] LustreError: 240846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448223 with bad export cookie 9823416526684783053 [ 7695.965369] LustreError: 240846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7695.966938] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7696.345732] Lustre: server umount lustre-MDT0001 complete [ 7705.613526] Lustre: server umount lustre-OST0000 complete [ 7716.883489] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 11:10:43 (1788448243) [ 7728.852689] Lustre: server umount lustre-OST0001 complete [ 7736.379751] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [ 7747.636352] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 7785.755663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7786.014842] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7786.028338] Lustre: Skipped 2 previous similar messages [ 7786.070848] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7786.170686] Lustre: lustre-MDT0000: new disk, initializing [ 7786.246955] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7786.270158] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7790.591857] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 7800.904824] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7801.009632] Lustre: 248363:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7801.075909] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7801.079722] Lustre: Skipped 1 previous similar message [ 7801.142771] Lustre: lustre-MDT0001: new disk, initializing [ 7801.279476] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7801.316111] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7801.326231] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7805.775849] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 7815.004714] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7820.332719] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7827.652242] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7827.918650] Lustre: lustre-OST0000: new disk, initializing [ 7827.923107] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7827.932534] Lustre: 250333:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7828.024167] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7836.855685] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 7838.309179] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7838.353426] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7838.730187] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7850.816872] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7853.523490] LustreError: 251004:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7863.616095] LustreError: 251004:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7863.647093] LustreError: 251103:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7864.216203] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7864.340536] Lustre: lustre-OST0001: new disk, initializing [ 7864.343624] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7864.360492] Lustre: 251450:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7864.516341] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7871.167773] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7871.179275] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7871.362824] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7871.563635] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 7873.747966] LustreError: 251103:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7883.338760] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7886.057983] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7891.939204] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7891.941669] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7891.951656] LustreError: Skipped 1 previous similar message [ 7891.956589] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7891.971532] Lustre: Skipped 1 previous similar message [ 7891.987178] Lustre: Skipped 4 previous similar messages [ 7895.197729] Lustre: server umount lustre-OST0001 complete [ 7897.066792] LustreError: 250317:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7897.093745] LustreError: 250317:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7915.495926] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7915.496703] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7915.512368] LustreError: Skipped 1 previous similar message [ 7915.515956] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7915.529440] Lustre: Skipped 1 previous similar message [ 7915.549291] Lustre: Skipped 1 previous similar message [ 7919.350076] Lustre: server umount lustre-OST0000 complete [ 7924.192256] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7924.195630] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7924.230039] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7924.237691] Lustre: Skipped 2 previous similar messages [ 7930.246548] Lustre: server umount lustre-MDT0000 complete [ 7932.897247] LustreError: 251103:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7932.924453] LustreError: 251103:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 7934.045934] LustreError: 249160:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448461 with bad export cookie 9823416526684785650 [ 7934.050568] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7934.058233] LustreError: 249160:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7940.400855] Lustre: server umount lustre-MDT0001 complete [ 7952.437674] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 7954.804393] Key type lgssc unregistered [ 7955.129784] LNet: 253489:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7955.140114] LNetError: 253489:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7956.199846] LNet: Removed LNI 192.168.201.113@tcp [ 7957.098861] Key type .llcrypt unregistered [ 7957.101097] Key type ._llcrypt unregistered [ 7995.854659] Key type ._llcrypt registered [ 7995.856909] Key type .llcrypt registered [ 7995.930519] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 7996.845608] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7996.905873] alg: No test for adler32 (adler32-zlib) [ 7997.908220] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 7998.114946] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 7999.759181] Key type lgssc registered [ 8000.758705] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8007.693738] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8007.711732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8008.926981] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8008.961494] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8009.121678] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8013.166707] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 8021.360312] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8021.480732] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8021.500161] Lustre: 255689:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8021.684172] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8025.948530] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 8033.238822] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8038.960685] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8045.424401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8045.628617] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8045.821824] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8051.667635] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 8051.907791] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8059.235569] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8113.638949] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8113.644939] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8113.678251] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8113.701371] Lustre: Skipped 1 previous similar message [ 8116.517145] Lustre: server umount lustre-OST0000 complete [ 8120.931301] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8133.778599] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8134.153680] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8139.662514] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 8147.764573] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8153.535571] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8153.545839] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8153.549698] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8153.578608] Lustre: Skipped 1 previous similar message [ 8189.410674] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8189.414733] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8189.415914] LustreError: Skipped 1 previous similar message [ 8189.416923] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8189.445795] Lustre: Skipped 2 previous similar messages [ 8193.905255] Lustre: server umount lustre-OST0000 complete [ 8199.649385] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8199.657042] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8199.662539] Lustre: Skipped 1 previous similar message [ 8199.671258] Lustre: Skipped 2 previous similar messages [ 8204.415534] Lustre: server umount lustre-MDT0000 complete [ 8209.090076] LustreError: 255684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448736 with bad export cookie 4281369988127557480 [ 8209.096711] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8209.110730] LustreError: 255684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8209.607194] Lustre: server umount lustre-MDT0001 complete [ 8220.853171] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 8223.818499] Key type lgssc unregistered [ 8224.119976] LNet: 261114:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8224.125575] LNetError: 261114:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8225.193840] LNet: Removed LNI 192.168.201.113@tcp [ 8226.510994] Key type .llcrypt unregistered [ 8226.513087] Key type ._llcrypt unregistered [ 8250.005749] Key type ._llcrypt registered [ 8250.009514] Key type .llcrypt registered [ 8250.103090] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 8251.421790] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8251.457599] alg: No test for adler32 (adler32-zlib) [ 8252.511831] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 8252.770096] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 8254.479556] Key type lgssc registered [ 8255.411296] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8262.488259] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8262.528787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8264.065845] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8268.626557] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 8274.402295] LustreError: 262536:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8274.416570] LustreError: 262536:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8277.624734] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8278.009539] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8282.175431] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 8290.027319] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8295.885909] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8303.245725] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8303.591192] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 8303.731374] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8310.268684] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8310.611460] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 8318.605330] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8345.056647] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8345.077093] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8345.105555] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8346.090982] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8346.109809] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8346.121704] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8346.127313] Lustre: Skipped 1 previous similar message [ 8348.808334] Lustre: server umount lustre-OST0000 complete [ 8354.789515] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8354.799188] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8354.830949] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8358.905313] Lustre: server umount lustre-MDT0000 complete [ 8361.448249] LustreError: 262530:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8361.471670] LustreError: 262530:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8362.752058] LustreError: 265887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448889 with bad export cookie 4032349624999849499 [ 8362.754748] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8362.764137] LustreError: 265887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8363.297576] Lustre: server umount lustre-MDT0001 complete [ 8375.199794] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 8378.007714] Key type lgssc unregistered [ 8378.342344] LNet: 266843:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8378.356662] LNetError: 266843:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8378.368805] LNet: Removed LNI 192.168.201.113@tcp [ 8379.278271] Key type .llcrypt unregistered [ 8379.279962] Key type ._llcrypt unregistered [ 8397.823230] Key type ._llcrypt registered [ 8397.825591] Key type .llcrypt registered [ 8397.932190] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 8399.132621] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8399.149836] alg: No test for adler32 (adler32-zlib) [ 8400.336319] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 8400.680776] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 8402.440523] Key type lgssc registered [ 8403.637392] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8411.489661] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8411.513762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8412.923604] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8417.800261] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 8423.401791] LustreError: 268063:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8423.425944] LustreError: 268063:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8426.318735] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8426.654742] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8430.469782] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 8437.996850] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8443.425780] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8451.619489] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8451.927058] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8458.094089] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8459.297931] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 8468.221675] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8483.814189] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8483.831613] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8483.856711] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8486.896712] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8486.904093] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8486.912294] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8489.653171] Lustre: server umount lustre-OST0000 complete [ 8497.123117] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8497.141439] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8497.154328] Lustre: Skipped 2 previous similar messages [ 8502.240835] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8502.250389] Lustre: Skipped 1 previous similar message [ 8507.379201] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8507.384061] Lustre: Skipped 1 previous similar message [ 8508.383681] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8508.504421] Lustre: server umount lustre-MDT0000 complete [ 8512.118408] LustreError: 270789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449039 with bad export cookie 3878528862882064166 [ 8512.132204] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8512.137413] LustreError: 270789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8512.485537] Lustre: server umount lustre-MDT0001 complete [ 8525.016885] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 8528.711285] Key type lgssc unregistered [ 8528.947855] LNet: 271748:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8528.957995] LNetError: 271748:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8528.972813] LNet: Removed LNI 192.168.201.113@tcp [ 8529.799288] Key type .llcrypt unregistered [ 8529.801477] Key type ._llcrypt unregistered [ 8552.817157] Key type ._llcrypt registered [ 8552.820655] Key type .llcrypt registered [ 8552.927148] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 8554.104309] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8554.316955] alg: No test for adler32 (adler32-zlib) [ 8555.447385] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 8555.754203] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 8557.479402] Key type lgssc registered [ 8558.400933] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8568.240332] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8568.259798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8569.855549] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8575.457485] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 8579.108652] LustreError: 273167:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8579.131738] LustreError: 273167:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8584.167443] LustreError: 273168:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8585.752237] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8586.226717] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8590.607038] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 8597.909704] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8603.766067] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8610.568895] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8610.804659] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8613.952073] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8617.213550] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 8625.403742] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8652.770733] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8652.778913] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8652.792914] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8654.831669] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8654.832803] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8654.833088] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8657.056690] Lustre: server umount lustre-OST0000 complete [ 8663.011515] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8663.016147] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8663.025160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8663.027740] Lustre: Skipped 1 previous similar message [ 8665.059592] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8665.062522] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8666.963495] Lustre: server umount lustre-MDT0000 complete [ 8670.180753] LustreError: 275177:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8670.211334] LustreError: 275177:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8670.766342] LustreError: 273149:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449197 with bad export cookie 7076286528641391918 [ 8670.775057] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8670.777639] LustreError: 273149:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8671.100869] Lustre: server umount lustre-MDT0001 complete [ 8682.715534] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 8685.588503] Key type lgssc unregistered [ 8685.972682] LNet: 277477:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8685.984512] LNetError: 277477:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8687.017635] LNet: Removed LNI 192.168.201.113@tcp [ 8688.067255] Key type .llcrypt unregistered [ 8688.073586] Key type ._llcrypt unregistered [ 8705.799692] Key type ._llcrypt registered [ 8705.802135] Key type .llcrypt registered [ 8705.884858] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 8707.631781] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8707.692929] alg: No test for adler32 (adler32-zlib) [ 8708.821781] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 8709.164707] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 8710.943209] Key type lgssc registered [ 8712.346575] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8721.744927] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8721.783261] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8723.437953] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8728.888433] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 8733.668539] LustreError: 278717:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8733.681791] LustreError: 278717:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8738.301285] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8738.577815] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8743.651298] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 8752.461911] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8759.217835] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8766.709767] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8767.065664] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8768.122293] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8772.200647] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 8779.883454] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8799.203452] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8799.209924] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8799.213237] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8799.253346] Lustre: Skipped 1 previous similar message [ 8800.344610] Lustre: server umount lustre-OST0000 complete [ 8804.334301] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8804.351614] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8804.360360] Lustre: Skipped 1 previous similar message [ 8804.369686] Lustre: Skipped 1 previous similar message [ 8809.441305] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8809.464674] Lustre: Skipped 2 previous similar messages [ 8810.095717] Lustre: server umount lustre-MDT0000 complete [ 8814.008041] LustreError: 278682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449341 with bad export cookie 14064987438060442892 [ 8814.012456] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8814.024916] LustreError: 278682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8814.266162] Lustre: server umount lustre-MDT0001 complete [ 8825.017989] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 8828.129752] Key type lgssc unregistered [ 8828.512592] LNet: 282418:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8828.516359] LNetError: 282418:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8829.543862] LNet: Removed LNI 192.168.201.113@tcp [ 8830.898496] Key type .llcrypt unregistered [ 8830.900446] Key type ._llcrypt unregistered [ 8843.855696] Key type ._llcrypt registered [ 8843.857596] Key type .llcrypt registered [ 8843.970870] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8861.590568] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [ 8878.797295] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 8879.816389] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8879.955065] alg: No test for adler32 (adler32-zlib) [ 8881.004387] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 8881.220462] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 8882.903215] Key type lgssc registered [ 8884.049674] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8921.609000] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8921.630156] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8922.862802] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8922.886861] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8922.943481] Lustre: lustre-MDT0000: new disk, initializing [ 8923.000189] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8923.012825] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8926.944413] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 8936.753084] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8936.930641] Lustre: 285972:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8936.980609] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8936.984555] Lustre: Skipped 1 previous similar message [ 8937.052250] Lustre: lustre-MDT0001: new disk, initializing [ 8937.194388] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8937.239815] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8937.249482] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8941.564749] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 8949.850785] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8954.777160] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8961.705579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8961.984968] Lustre: lustre-OST0000: new disk, initializing [ 8961.993431] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8962.004866] Lustre: 287939:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8962.125694] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8967.787568] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 8970.323646] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8970.336616] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8970.468648] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8977.235414] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8981.057447] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8981.194159] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8985.834823] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8986.189598] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8991.200272] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8991.201630] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8991.203249] LustreError: Skipped 1 previous similar message [ 8991.215075] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8991.228218] Lustre: Skipped 1 previous similar message [ 8994.470364] Lustre: server umount lustre-OST0000 complete [ 8998.894583] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8998.905709] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8998.949960] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8998.961557] Lustre: Skipped 1 previous similar message [ 9001.441501] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9001.445637] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9005.151084] Lustre: server umount lustre-MDT0000 complete [ 9009.408352] LustreError: 285966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449536 with bad export cookie 6449729881667341594 [ 9009.415516] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9009.425731] LustreError: 285966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9009.891386] Lustre: server umount lustre-MDT0001 complete [ 9019.365951] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 11:32:25 (1788449545) [ 9026.436269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9026.954355] LustreError: 290350:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9026.973600] LustreError: 290350:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9027.062366] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9031.700142] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 9032.161740] LustreError: 290350:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9032.197650] LustreError: 290350:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9037.284138] LustreError: 290351:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9042.165048] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9042.405438] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 9042.420697] Lustre: Skipped 1 previous similar message [ 9042.639456] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9048.634049] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 9058.194171] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9065.695190] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9073.787674] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9074.325401] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9080.449684] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 9087.934729] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9107.441520] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9107.449344] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9107.462642] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9107.483340] Lustre: Skipped 1 previous similar message [ 9112.573888] Lustre: server umount lustre-OST0000 complete [ 9117.677830] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9117.684694] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9117.696857] Lustre: Skipped 2 previous similar messages [ 9117.712894] Lustre: Skipped 1 previous similar message [ 9122.605592] Lustre: server umount lustre-MDT0000 complete [ 9122.786714] LustreError: 290351:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9122.809749] LustreError: 290351:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9126.217882] LustreError: 290331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449653 with bad export cookie 6449729881667342476 [ 9126.221971] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9126.227382] LustreError: 290331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9126.492770] Lustre: server umount lustre-MDT0001 complete [ 9137.719237] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 9140.683474] Key type lgssc unregistered [ 9140.941222] LNet: 293915:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9140.950651] LNetError: 293915:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9141.995145] LNet: Removed LNI 192.168.201.113@tcp [ 9142.966854] Key type .llcrypt unregistered [ 9142.968870] Key type ._llcrypt unregistered [ 9172.365597] Key type ._llcrypt registered [ 9172.371680] Key type .llcrypt registered [ 9172.560527] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [ 9195.441529] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 9196.712773] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9196.752459] alg: No test for adler32 (adler32-zlib) [ 9198.369300] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 9198.750183] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 9200.415176] Key type lgssc registered [ 9202.373643] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9257.804598] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9257.859352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9259.268685] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9259.307442] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9259.383646] Lustre: lustre-MDT0000: new disk, initializing [ 9259.449590] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9259.461667] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9265.075643] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 9278.326680] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9278.464184] Lustre: 297746:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9278.518519] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9278.522764] Lustre: Skipped 1 previous similar message [ 9278.600855] Lustre: lustre-MDT0001: new disk, initializing [ 9278.758259] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9278.824183] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9278.848911] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9284.845660] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 9296.870208] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9305.140510] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9313.892860] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9314.174705] Lustre: lustre-OST2710: new disk, initializing [ 9314.178222] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9314.187318] Lustre: 299713:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9314.262191] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9322.253518] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 9324.149545] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9324.170487] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9324.330978] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9332.518761] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9339.184702] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9339.460613] Lustre: lustre-OST03e8: new disk, initializing [ 9339.468080] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9339.478429] Lustre: 300779:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9339.601829] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9346.898915] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 9348.666843] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9348.674932] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9348.771501] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9355.203597] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9364.955220] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9365.318748] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9370.505673] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9370.836516] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9376.409930] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9376.660662] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9381.589325] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9381.919300] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9386.467299] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9386.478319] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9386.490397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9388.808063] Lustre: server umount lustre-MDT0000 complete [ 9390.060859] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9392.758068] LustreError: 297737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449919 with bad export cookie 6048826900195041175 [ 9392.773671] LustreError: 297737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9392.778914] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9393.284523] Lustre: server umount lustre-MDT0001 complete [ 9407.662807] Lustre: server umount lustre-OST2710 complete [ 9421.467215] Lustre: server umount lustre-OST03e8 complete [ 9441.328468] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [ 9453.709502] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 9492.490412] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9493.147218] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9493.213320] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9493.312976] Lustre: lustre-MDT0000: new disk, initializing [ 9493.399302] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9493.419305] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9498.479694] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 9511.413089] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9511.671710] Lustre: 306032:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9511.755490] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9511.758878] Lustre: Skipped 1 previous similar message [ 9511.894775] Lustre: lustre-MDT0001: new disk, initializing [ 9511.998736] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9512.038065] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9512.049859] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9517.704047] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 9528.386773] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9535.350724] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9542.500522] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9542.788000] Lustre: lustre-OST0000: new disk, initializing [ 9542.791484] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9542.798552] Lustre: 308000:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9542.896396] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9548.406118] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9548.435714] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9548.592440] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9550.130140] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 9561.317409] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9566.391829] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9566.625518] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9571.632362] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9571.876710] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9574.373166] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9574.374446] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9574.383330] Lustre: Skipped 2 previous similar messages [ 9574.391175] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9574.415285] Lustre: Skipped 1 previous similar message [ 9579.496760] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9579.507878] Lustre: Skipped 1 previous similar message [ 9580.222524] Lustre: server umount lustre-OST0000 complete [ 9584.609357] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9584.619782] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9584.622049] Lustre: Skipped 2 previous similar messages [ 9589.732249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9589.743927] Lustre: Skipped 1 previous similar message [ 9590.191301] Lustre: server umount lustre-MDT0000 complete [ 9594.355401] LustreError: 306023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450121 with bad export cookie 6048826900195042855 [ 9594.357598] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9594.368899] LustreError: 306023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9594.876213] Lustre: server umount lustre-MDT0001 complete [ 9604.836476] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 11:42:11 (1788450131) [ 9606.453478] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9608.656349] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 11:42:15 (1788450135) [ 9613.179480] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 9640.501520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9640.827113] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9640.883964] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9641.061983] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9645.267469] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 9653.645579] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9653.838755] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9653.889827] Lustre: 311858:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9658.445306] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 9679.789344] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9680.093923] Lustre: Denying initial registration attempt from nid 192.168.201.113@tcp, specified as failover [ 9680.108730] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9680.119208] LustreError: 313312:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 9680.125195] LustreError: 313312:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9680.132496] LustreError: 313312:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9680.238589] Lustre: server umount lustre-OST0000 complete [ 9680.241986] LustreError: 313312:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9687.522588] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9687.533355] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9687.539537] Lustre: Skipped 1 previous similar message [ 9687.559689] Lustre: Skipped 1 previous similar message [ 9700.322192] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9700.685448] Lustre: server umount lustre-MDT0000 complete [ 9702.883214] LustreError: 311871:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9702.903353] LustreError: 311871:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 9704.764785] LustreError: 311851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450231 with bad export cookie 6048826900195043737 [ 9704.768847] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9704.771933] LustreError: 311851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9705.127886] Lustre: server umount lustre-MDT0001 complete [ 9715.634396] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 9718.548356] Key type lgssc unregistered [ 9718.773028] LNet: 314438:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9718.781694] LNetError: 314438:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9718.798752] LNet: Removed LNI 192.168.201.113@tcp [ 9719.877251] Key type .llcrypt unregistered [ 9719.879297] Key type ._llcrypt unregistered [ 9733.623664] Key type ._llcrypt registered [ 9733.629366] Key type .llcrypt registered [ 9733.775705] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 9734.761983] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9734.958763] alg: No test for adler32 (adler32-zlib) [ 9736.105624] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 9736.492137] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 9738.263185] Key type lgssc registered [ 9739.643144] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9788.966850] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 9797.668949] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9797.691404] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9798.952815] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9798.998208] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9799.162652] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9803.914883] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 9813.058899] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9813.329558] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9813.392907] Lustre: 317203:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9813.730791] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9818.659925] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 9827.609105] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9834.603384] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9854.703584] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9854.910399] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9855.105107] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9861.530091] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 9870.104420] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9875.438332] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9875.458435] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9875.502787] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9878.499219] Lustre: server umount lustre-OST0000 complete [ 9885.668630] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9885.669168] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9885.671066] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9885.683237] Lustre: Skipped 2 previous similar messages [ 9885.694069] Lustre: Skipped 1 previous similar message [ 9888.706521] Lustre: server umount lustre-MDT0000 complete [ 9892.654158] LustreError: 317196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450419 with bad export cookie 9207741512765481736 [ 9892.664134] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9892.669233] LustreError: 317196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9893.286468] Lustre: server umount lustre-MDT0001 complete [ 9904.661496] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 9907.334831] Key type lgssc unregistered [ 9907.652102] LNet: 320783:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9907.663952] LNetError: 320783:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9907.680876] LNet: Removed LNI 192.168.201.113@tcp [ 9908.645447] Key type .llcrypt unregistered [ 9908.649285] Key type ._llcrypt unregistered [ 9932.319421] Key type ._llcrypt registered [ 9932.321427] Key type .llcrypt registered [ 9932.390367] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 9933.492854] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9933.552743] alg: No test for adler32 (adler32-zlib) [ 9934.550834] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 9934.734731] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 9936.367188] Key type lgssc registered [ 9937.388335] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9944.295266] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9944.309681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9945.745361] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9950.167459] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 9955.300827] LustreError: 322200:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9955.325891] LustreError: 322200:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9959.059034] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9959.429777] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9964.101162] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 9971.902730] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9977.497336] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9985.405707] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9985.674734] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9991.259636] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 9998.622775] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10015.732570] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10015.766386] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10015.793457] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10016.731599] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.13@tcp (stopping) [10017.342865] Lustre: server umount lustre-MDT0000 complete [10019.811308] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10019.813272] LustreError: 322201:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10019.831423] Lustre: Skipped 1 previous similar message [10019.866161] LustreError: 322201:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10021.610254] LustreError: 322181:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450548 with bad export cookie 10337190876040170459 [10021.620376] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10021.624399] LustreError: 322181:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10021.815420] LustreError: 323440:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10021.823506] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.13@tcp (stopping) [10024.929454] LustreError: 322197:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10024.935895] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10024.985197] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10028.129423] Lustre: server umount lustre-MDT0001 complete [10034.850650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10043.562199] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10046.433735] LustreError: 321509:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000001ff6661e x1875326230634496/t0(0) o250->MGC192.168.201.113@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10046.708310] LustreError: 324032:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10046.723371] LustreError: 324032:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10046.802913] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10046.877293] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [10050.590189] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [10055.971417] LustreError: 325814:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10055.974987] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10056.006376] LustreError: 325814:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [10058.826483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10059.311550] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10064.271229] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [10064.360486] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10071.925304] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10073.478814] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10079.487274] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10081.071905] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10087.915286] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10087.929480] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10087.940332] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10087.943087] Lustre: Skipped 2 previous similar messages [10089.958067] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10091.448911] Lustre: server umount lustre-OST0000 complete [10095.075202] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10095.075537] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10095.092731] Lustre: Skipped 1 previous similar message [10100.193129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10100.197575] Lustre: Skipped 5 previous similar messages [10109.413237] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10109.858734] Lustre: server umount lustre-MDT0000 complete [10110.434660] LustreError: 325819:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10110.459919] LustreError: 325819:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10113.929575] LustreError: 322181:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450641 with bad export cookie 10337190876040178047 [10113.943566] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10113.944863] LustreError: 322181:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10114.247539] Lustre: server umount lustre-MDT0001 complete [10126.732806] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [10129.294511] Key type lgssc unregistered [10129.576208] LNet: 328495:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10129.579540] LNetError: 328495:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10129.596707] LNet: Removed LNI 192.168.201.113@tcp [10130.403640] Key type .llcrypt unregistered [10130.405954] Key type ._llcrypt unregistered [10155.271617] Key type ._llcrypt registered [10155.273674] Key type .llcrypt registered [10155.401530] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [10156.455607] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10156.489354] alg: No test for adler32 (adler32-zlib) [10157.669249] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [10157.876399] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [10159.527199] Key type lgssc registered [10160.980423] Lustre: Echo OBD driver; http://www.lustre.org/ [10169.007640] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10169.037458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10170.608826] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10176.371129] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [10180.081390] LustreError: 329912:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10180.096344] LustreError: 329912:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10185.184501] LustreError: 329913:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10185.399668] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10185.873863] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10190.485982] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [10197.983776] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10204.324849] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10210.596040] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10210.601730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10210.626181] Lustre: Skipped 1 previous similar message [10213.927271] Lustre: server umount lustre-MDT0000 complete [10215.905737] LustreError: 329913:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10215.915624] LustreError: 329913:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10217.833938] LustreError: 329891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450744 with bad export cookie 3553264957058143939 [10217.843737] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10217.849033] LustreError: 329891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10218.323294] Lustre: server umount lustre-MDT0001 complete [10233.478293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10233.748201] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10233.784944] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10233.970733] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10238.232171] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [10247.470121] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10247.571067] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10247.608494] Lustre: 332500:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10247.919924] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10252.230145] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [10258.869846] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10264.624526] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10274.806605] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10275.109628] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10275.123703] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10275.137882] LustreError: 334386:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [10275.149909] LustreError: 334386:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [10275.164367] LustreError: 334386:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10275.254432] Lustre: server umount lustre-OST0000 complete [10275.258105] LustreError: 334386:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [10286.647063] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10286.889710] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10287.093355] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10292.868898] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10293.472566] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [10301.862383] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10310.453841] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10310.561746] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10310.653796] Lustre: lustre-OST0001: new disk, initializing [10310.657898] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10310.664784] Lustre: 335925:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10310.795755] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10316.665951] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [10316.901211] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10316.918159] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10317.029073] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10323.427605] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10327.521335] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10327.535758] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10327.565226] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10327.571114] Lustre: Skipped 2 previous similar messages [10332.652186] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10332.660504] Lustre: Skipped 1 previous similar message [10337.770576] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10337.779842] Lustre: Skipped 1 previous similar message [10340.319163] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10340.447427] Lustre: server umount lustre-OST0001 complete [10342.904305] LustreError: 334851:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10342.938422] LustreError: 334851:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10344.421886] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10344.434542] LustreError: Skipped 1 previous similar message [10344.441780] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10344.462742] Lustre: Skipped 1 previous similar message [10344.472930] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10348.015744] LustreError: 336303:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10348.019939] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10348.027146] LustreError: 336303:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10350.464731] Lustre: server umount lustre-OST0000 complete [10355.681097] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10355.709579] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10355.730290] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10355.737920] Lustre: Skipped 2 previous similar messages [10361.619200] Lustre: server umount lustre-MDT0000 complete [10363.366096] LustreError: 334102:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10363.380313] LustreError: 334102:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10365.483502] LustreError: 332492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450892 with bad export cookie 3553264957058144492 [10365.488271] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10365.506399] LustreError: 332492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10366.135616] Lustre: server umount lustre-MDT0001 complete [10378.628780] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [10381.473568] Key type lgssc unregistered [10381.712613] LNet: 337890:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10381.717317] LNetError: 337890:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10382.774417] LNet: Removed LNI 192.168.201.113@tcp [10383.571496] Key type .llcrypt unregistered [10383.573110] Key type ._llcrypt unregistered [10454.241472] Key type ._llcrypt registered [10454.244166] Key type .llcrypt registered [10454.331219] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [10469.503642] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [10470.447778] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10470.490749] alg: No test for adler32 (adler32-zlib) [10471.438778] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [10471.639561] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [10473.279391] Key type lgssc registered [10474.242282] Lustre: Echo OBD driver; http://www.lustre.org/ [10504.577523] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10504.599424] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10505.826760] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10505.847188] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10505.912739] Lustre: lustre-MDT0000: new disk, initializing [10505.986664] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10505.999974] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10509.775989] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [10519.778186] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10519.917079] Lustre: 342824:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10519.966541] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10519.970148] Lustre: Skipped 1 previous similar message [10520.030409] Lustre: lustre-MDT0001: new disk, initializing [10520.124720] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10520.164163] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10520.183129] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10524.772957] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [10533.700743] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10539.708924] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10546.549755] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10546.841812] Lustre: lustre-OST0000: new disk, initializing [10546.844296] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10546.851179] Lustre: 344793:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10546.940665] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10552.420747] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10552.426840] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10552.610615] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10553.996603] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [10565.294606] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10570.010495] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10570.204985] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10576.861455] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10577.100588] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10583.522864] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10583.529401] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10583.577210] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10585.451291] Lustre: server umount lustre-OST0000 complete [10591.713456] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10591.723523] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10591.742354] Lustre: Skipped 1 previous similar message [10591.749498] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10591.755203] Lustre: Skipped 1 previous similar message [10593.762532] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10593.765091] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10595.129068] Lustre: server umount lustre-MDT0000 complete [10598.938987] LustreError: 343627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451126 with bad export cookie 12352662267067357226 [10598.950360] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10599.620613] Lustre: server umount lustre-MDT0001 complete [10609.188820] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 11:58:55 (1788451135) [10617.236654] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 11:59:03 (1788451143) [10625.920377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10626.412555] LustreError: 347547:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10626.441705] LustreError: 347547:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10626.541996] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10631.649873] LustreError: 347548:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10632.044832] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [10636.768570] LustreError: 347547:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10641.890476] LustreError: 347548:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10642.460224] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10642.936334] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10647.714750] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [10656.590272] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10662.720595] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10669.461829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10669.759308] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10674.812888] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [10682.772857] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10693.074544] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10694.676738] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10696.609981] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10698.210859] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10700.062609] Lustre: Failing over lustre-MDT0000 [10700.244777] Lustre: server umount lustre-MDT0000 complete [10703.840850] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10703.848778] LustreError: 347544:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10703.850162] Lustre: Skipped 1 previous similar message [10703.878321] LustreError: 347544:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10714.082709] LustreError: 347542:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10714.113716] LustreError: 347542:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [10717.807793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10717.894641] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10718.058872] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10718.104251] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10720.109251] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10722.097460] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [10723.316422] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10723.369901] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [10723.423804] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10732.562116] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10735.021740] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10739.908832] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10742.301143] Lustre: Failing over lustre-MDT0000 [10742.674270] Lustre: server umount lustre-MDT0000 complete [10743.780721] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10743.785760] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10743.788911] LustreError: 348546:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10743.788918] LustreError: 348546:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10743.861402] Lustre: Skipped 2 previous similar messages [10759.137262] Lustre: 341050:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788451270/real 1788451270] req@00000000ac26b359 x1875326793771904/t0(0) o400->MGC192.168.201.113@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788451286 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10759.161781] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10763.816107] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10769.740045] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10769.801504] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10772.410431] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10774.439063] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [10775.025503] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10775.039649] Lustre: Skipped 2 previous similar messages [10775.071525] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [10775.138739] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10783.957827] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10785.819044] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10795.489476] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10795.493674] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10795.499629] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10795.499641] Lustre: Skipped 1 previous similar message [10795.536229] Lustre: Skipped 3 previous similar messages [10798.730342] Lustre: server umount lustre-MDT0000 complete [10800.610866] LustreError: 347542:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10800.635954] LustreError: 347542:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 25 previous similar messages [10803.391440] LustreError: 347529:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451330 with bad export cookie 12352662267067359935 [10803.405292] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10803.408223] LustreError: 347529:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [10803.756104] Lustre: server umount lustre-MDT0001 complete [10819.024754] Lustre: server umount lustre-OST0000 complete [10835.861756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10836.360631] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10840.996598] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [10850.190420] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10854.914043] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [10863.070133] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10868.922189] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10875.589296] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10881.112713] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10881.831784] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [10887.815525] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10905.064123] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [10905.077375] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10905.091744] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10905.103646] Lustre: Skipped 2 previous similar messages [10909.776234] Lustre: server umount lustre-OST0000 complete [10915.809575] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10915.818577] Lustre: Skipped 2 previous similar messages [10919.475227] Lustre: server umount lustre-MDT0000 complete [10920.929356] LustreError: 353898:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10920.950771] LustreError: 353898:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [10922.780623] LustreError: 353877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451449 with bad export cookie 12352662267067360684 [10922.791246] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10922.796973] LustreError: 353877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10923.295417] Lustre: server umount lustre-MDT0001 complete [10933.833979] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [10936.544474] Key type lgssc unregistered [10936.836591] LNet: 357452:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10936.847485] LNetError: 357452:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10936.858575] LNet: Removed LNI 192.168.201.113@tcp [10937.470398] Key type .llcrypt unregistered [10937.475307] Key type ._llcrypt unregistered [10965.621932] Key type ._llcrypt registered [10965.630497] Key type .llcrypt registered [10965.771623] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [10966.812842] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10966.977593] alg: No test for adler32 (adler32-zlib) [10968.042607] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [10968.321396] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [10970.063190] Key type lgssc registered [10971.157712] Lustre: Echo OBD driver; http://www.lustre.org/ [10979.346757] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10979.374021] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10980.912189] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10985.695745] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [10991.077132] LustreError: 358971:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10991.116392] LustreError: 358971:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10994.756301] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10995.233612] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10999.895674] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [11007.994991] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11014.464591] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11022.133538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11022.502967] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11026.637827] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11030.032474] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [11038.170836] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11051.488601] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11051.500429] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11051.517857] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11055.587925] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11055.597800] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11055.606859] Lustre: Skipped 2 previous similar messages [11057.558893] Lustre: server umount lustre-MDT0000 complete [11060.706665] LustreError: 359973:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11060.718933] LustreError: 359973:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11061.311886] LustreError: 360814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451588 with bad export cookie 11337614917228993981 [11061.317977] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11061.325793] LustreError: 360814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11061.612137] Lustre: server umount lustre-MDT0001 complete [11075.505106] Lustre: server umount lustre-OST0000 complete [11090.455585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11090.972335] LustreError: 362703:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11091.049768] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11096.016353] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [11096.546071] LustreError: 362704:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11096.581473] LustreError: 362704:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11101.673841] LustreError: 362703:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11104.423350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11104.833015] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11109.185183] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [11117.502982] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11123.270670] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11130.370732] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11130.765799] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11135.928243] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11137.037478] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [11145.362854] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11159.534277] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11159.539310] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11159.552794] Lustre: Skipped 3 previous similar messages [11159.560584] Lustre: Skipped 2 previous similar messages [11164.626191] Lustre: server umount lustre-MDT0000 complete [11164.641469] LustreError: 363432:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11164.668141] LustreError: 363432:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11168.696448] LustreError: 362685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451695 with bad export cookie 11337614917228995101 [11168.708026] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11168.708334] LustreError: 362685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11169.317039] Lustre: server umount lustre-MDT0001 complete [11184.009734] Lustre: server umount lustre-OST0000 complete [11202.133943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11202.564504] LustreError: 366616:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11202.688047] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11207.554995] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [11216.409395] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11221.790234] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [11229.397302] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11236.872833] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11243.421903] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11243.686746] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11243.691376] Lustre: Skipped 1 previous similar message [11247.807511] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11249.202630] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [11256.606225] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11263.462736] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11263.474145] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11263.497785] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11268.630918] Lustre: server umount lustre-OST0000 complete [11273.185114] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11273.190319] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11273.197901] Lustre: Skipped 1 previous similar message [11273.199809] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11273.206451] Lustre: Skipped 2 previous similar messages [11279.006896] Lustre: server umount lustre-MDT0000 complete [11281.889751] LustreError: 367345:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11281.906038] LustreError: 367345:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11283.065921] LustreError: 366598:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451810 with bad export cookie 11337614917228996123 [11283.072981] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11283.093712] LustreError: 366598:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11283.534858] Lustre: server umount lustre-MDT0001 complete [11295.981414] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [11298.971862] Key type lgssc unregistered [11299.229950] LNet: 370152:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11299.234751] LNetError: 370152:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11299.245940] LNet: Removed LNI 192.168.201.113@tcp [11300.137837] Key type .llcrypt unregistered [11300.139808] Key type ._llcrypt unregistered [11328.355351] Key type ._llcrypt registered [11328.358705] Key type .llcrypt registered [11328.471076] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [11329.833853] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11329.929709] alg: No test for adler32 (adler32-zlib) [11331.143423] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [11331.469680] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [11333.135266] Key type lgssc registered [11334.213525] Lustre: Echo OBD driver; http://www.lustre.org/ [11342.664937] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11342.685268] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11342.702444] LustreError: 371593:0:(osd_handler.c:8554:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11342.725257] LustreError: 371593:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11342.732252] LustreError: 371593:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11342.748219] LustreError: 371593:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11342.761163] LustreError: 371593:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [11350.590902] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11350.595635] LustreError: 371990:0:(osd_handler.c:8554:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11350.616692] LustreError: 371990:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11350.631912] LustreError: 371990:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11350.644838] LustreError: 371990:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11350.650508] LustreError: 371990:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [11373.887375] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [11376.758505] Key type lgssc unregistered [11377.143947] LNet: 372997:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11377.149700] LNetError: 372997:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11377.163739] LNet: Removed LNI 192.168.201.113@tcp [11377.907531] Key type .llcrypt unregistered [11377.917611] Key type ._llcrypt unregistered [11397.731316] Key type ._llcrypt registered [11397.733317] Key type .llcrypt registered [11397.794143] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [11413.633633] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [11414.943199] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11414.953312] alg: No test for adler32 (adler32-zlib) [11416.002272] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [11416.203538] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [11417.831184] Key type lgssc registered [11419.428846] Lustre: Echo OBD driver; http://www.lustre.org/ [11455.839918] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11455.870089] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11457.144150] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11457.166715] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11457.222244] Lustre: lustre-MDT0000: new disk, initializing [11457.276089] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11457.289618] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11461.965544] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [11473.089422] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11473.232898] Lustre: 376155:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11473.272795] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11473.283075] Lustre: Skipped 1 previous similar message [11473.393772] Lustre: lustre-MDT0001: new disk, initializing [11473.520751] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11473.643893] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11473.653314] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11478.098737] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [11488.566424] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11495.602657] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11503.445612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11503.674149] Lustre: lustre-OST0000: new disk, initializing [11503.679981] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11503.691400] Lustre: 378126:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11503.766158] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11510.008286] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [11510.844719] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11510.856694] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11510.990058] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11520.228707] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11525.493430] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11525.716625] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11530.718508] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11530.966553] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11536.865399] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11536.874088] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11536.886142] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11541.988403] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11541.992572] Lustre: Skipped 2 previous similar messages [11547.103467] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11547.107048] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11547.129439] Lustre: Skipped 1 previous similar message [11547.414288] Lustre: server umount lustre-OST0000 complete [11552.225622] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11552.234318] Lustre: Skipped 2 previous similar messages [11552.237335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11552.243278] Lustre: Skipped 1 previous similar message [11557.346384] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11557.354976] Lustre: Skipped 1 previous similar message [11557.898630] Lustre: server umount lustre-MDT0000 complete [11562.784233] LustreError: 378125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452089 with bad export cookie 12429225391740448721 [11562.788288] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11562.796070] LustreError: 378125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11563.358783] Lustre: server umount lustre-MDT0001 complete [11573.413830] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11575.600421] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 12:15:01 (1788452101) [11583.081963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11583.427820] LustreError: 380589:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11583.446604] LustreError: 380589:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11583.513216] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11588.587976] LustreError: 380590:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11589.337797] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [11593.699738] LustreError: 380589:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11598.826989] LustreError: 380590:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11598.930775] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11599.258889] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11604.932874] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [11613.233713] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11619.749813] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11627.690239] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11627.938045] LustreError: 382412:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11628.143228] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11635.251605] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [11643.899298] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11651.588810] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11651.740598] Lustre: lustre-OST0001: new disk, initializing [11651.743586] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11651.751150] Lustre: 383426:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11651.832426] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11658.533578] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [11661.915098] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11661.932327] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11662.019348] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11668.982794] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11676.110590] Lustre: lustre-OST0001: Not available for connect from 192.168.201.13@tcp (stopping) [11677.665523] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11677.676598] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11677.692745] Lustre: Skipped 1 previous similar message [11677.715462] LustreError: Skipped 1 previous similar message [11678.997219] Lustre: server umount lustre-OST0001 complete [11681.201584] LustreError: 382414:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11698.146052] LustreError: 382412:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11698.153912] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11698.178713] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11698.178727] Lustre: Skipped 2 previous similar messages [11698.185776] LustreError: 382412:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [11698.222489] Lustre: Skipped 1 previous similar message [11703.268250] Lustre: server umount lustre-OST0000 complete [11708.235066] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11714.720311] Lustre: server umount lustre-MDT0000 complete [11718.947404] LustreError: 380570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452246 with bad export cookie 12429225391740449603 [11718.965076] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11718.966221] LustreError: 380570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11725.538281] Lustre: server umount lustre-MDT0001 complete [11736.747426] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [11740.244342] Key type lgssc unregistered [11740.644322] LNet: 385425:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11740.667528] LNetError: 385425:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11740.680250] LNet: Removed LNI 192.168.201.113@tcp [11741.709421] Key type .llcrypt unregistered [11741.711123] Key type ._llcrypt unregistered [11798.951602] Key type ._llcrypt registered [11798.956143] Key type .llcrypt registered [11799.106705] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [11800.951872] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11801.010179] alg: No test for adler32 (adler32-zlib) [11802.228092] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [11802.575246] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [11804.367518] Key type lgssc registered [11805.571497] Lustre: Echo OBD driver; http://www.lustre.org/ [11814.738570] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11814.753522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11815.982821] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11816.008962] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11816.156802] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11821.023118] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [11833.396419] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11833.482302] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11833.504942] Lustre: 388227:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11833.687401] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11839.297837] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [11849.044131] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11855.500431] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11859.430379] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11859.438479] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11859.454798] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11861.988497] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11861.990439] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11863.872070] Lustre: server umount lustre-MDT0000 complete [11867.641546] LustreError: 388221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452394 with bad export cookie 18019222204757643197 [11867.652175] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11867.660541] LustreError: 388221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11868.229471] Lustre: server umount lustre-MDT0001 complete [11878.961501] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 12:20:05 (1788452405) [11885.377189] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11885.866337] LustreError: 390742:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11885.878570] LustreError: 390742:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11885.939759] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11890.431946] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [11891.168609] LustreError: 390743:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11896.299234] LustreError: 390742:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11899.286131] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11899.676581] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11904.120694] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [11912.279860] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11919.669713] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11928.775503] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11929.026202] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11929.301058] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11936.449329] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [11945.806964] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11954.636634] Lustre: Permanently deactivating lustre-OST0000 [11954.649719] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11963.500634] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11965.203766] LustreError: 393727:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11965.212583] LustreError: 393727:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11965.218179] LustreError: 393727:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11968.190553] LustreError: 393775:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11968.197766] LustreError: 393775:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11976.341896] Lustre: server umount lustre-OST0000 complete [11979.169800] LustreError: 394025:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11979.173732] LustreError: 394025:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11981.794119] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11981.798617] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11981.810360] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11981.815643] Lustre: Skipped 1 previous similar message [11983.843386] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11983.845611] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11988.963347] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11988.979401] Lustre: Skipped 2 previous similar messages [11991.067670] Lustre: server umount lustre-MDT0000 complete [11994.082261] LustreError: 390742:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11995.305432] LustreError: 390723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452522 with bad export cookie 18019222204757643778 [11995.311538] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11995.321434] LustreError: 390723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12006.044390] Lustre: server umount lustre-MDT0001 complete [12016.321300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12021.760045] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [12025.092775] LustreError: 395186:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [12032.932661] Lustre: 395525:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.113@tcp [12035.855658] Lustre: server umount MGS complete [12048.433480] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12048.867378] LustreError: 396253:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12048.877670] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12048.878435] LustreError: 396253:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12048.885151] Lustre: Skipped 1 previous similar message [12048.924438] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12053.831753] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [12059.104565] LustreError: 396253:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12059.116767] LustreError: 396253:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12063.430205] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12063.864105] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12068.453841] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [12076.822368] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12083.017723] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12084.876415] Lustre: Permanently reactivating lustre-OST0000 [12084.891750] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12084.896448] Lustre: Skipped 2 previous similar messages [12112.863404] LustreError: 398032:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12123.446401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12123.924125] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12123.938258] Lustre: Skipped 1 previous similar message [12132.345494] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [12143.318431] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12152.801233] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12152.809181] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12152.829109] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12158.454590] Lustre: server umount lustre-OST0000 complete [12164.074810] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12164.105404] Lustre: Skipped 2 previous similar messages [12169.184609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12169.188936] Lustre: Skipped 5 previous similar messages [12177.376760] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12177.568880] Lustre: server umount lustre-MDT0000 complete [12179.424963] LustreError: 396248:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12179.442267] LustreError: 396248:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12181.603691] LustreError: 396233:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452708 with bad export cookie 18019222204757644828 [12181.630109] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12182.011609] Lustre: server umount lustre-MDT0001 complete [12193.418584] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [12196.305320] Key type lgssc unregistered [12196.559074] LNet: 400622:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12196.575452] LNetError: 400622:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12196.605769] LNet: Removed LNI 192.168.201.113@tcp [12197.494601] Key type .llcrypt unregistered [12197.496388] Key type ._llcrypt unregistered [12216.834505] Key type ._llcrypt registered [12216.836948] Key type .llcrypt registered [12216.975729] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [12230.809355] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [12231.659461] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12231.739296] alg: No test for adler32 (adler32-zlib) [12233.029792] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [12233.380735] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [12235.087207] Key type lgssc registered [12236.212820] Lustre: Echo OBD driver; http://www.lustre.org/ [12275.839113] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12275.874914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12277.386174] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12277.426223] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12277.567379] Lustre: lustre-MDT0000: new disk, initializing [12277.662278] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12277.691677] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12282.968781] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [12294.237757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12294.357608] Lustre: 403784:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12294.436825] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12294.440903] Lustre: Skipped 1 previous similar message [12294.525529] Lustre: lustre-MDT0001: new disk, initializing [12294.636089] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12294.683582] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12294.700449] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12300.101758] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [12310.486201] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12320.458564] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12331.132920] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12331.477064] Lustre: lustre-OST0000: new disk, initializing [12331.483937] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12331.490071] Lustre: 405751:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12331.667725] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12338.790006] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12338.802610] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12338.972417] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12339.722624] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [12351.694825] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12357.748838] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12357.930263] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12364.319744] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12364.546429] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12369.918067] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12369.931439] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12369.940918] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12369.985151] Lustre: Skipped 1 previous similar message [12373.504395] Lustre: server umount lustre-OST0000 complete [12380.129880] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12380.134033] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12380.158052] Lustre: Skipped 1 previous similar message [12380.169157] Lustre: Skipped 1 previous similar message [12385.250948] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12385.269137] Lustre: Skipped 1 previous similar message [12386.076679] Lustre: server umount lustre-MDT0000 complete [12390.588559] LustreError: 403774:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452917 with bad export cookie 889133818132863088 [12390.595038] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12390.605642] LustreError: 403774:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12390.934579] Lustre: server umount lustre-MDT0001 complete [12402.097216] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12404.350947] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 12:28:50 (1788452930) [12412.706050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12413.354422] LustreError: 408213:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12413.381136] LustreError: 408213:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12413.464272] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12418.535285] LustreError: 408214:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12418.559348] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [12423.651588] LustreError: 408213:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12428.016651] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12428.261203] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12433.089741] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [12442.174366] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12450.589841] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12460.631446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12461.089427] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12467.923684] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [12477.490352] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12479.841255] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12487.546523] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12493.792153] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12493.801310] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12493.802139] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12493.811190] LustreError: Skipped 1 previous similar message [12498.925791] LustreError: 410036:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12499.148783] Lustre: server umount lustre-OST0000 complete [12504.036479] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12504.045533] Lustre: Skipped 1 previous similar message [12504.050843] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12504.061567] Lustre: Skipped 2 previous similar messages [12509.640809] Lustre: server umount lustre-MDT0000 complete [12513.495567] LustreError: 408194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453040 with bad export cookie 889133818132863970 [12513.500674] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12513.505123] LustreError: 408194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12514.001536] Lustre: server umount lustre-MDT0001 complete [12526.524903] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [12530.589229] Key type lgssc unregistered [12530.934218] LNet: 411944:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12530.940394] LNetError: 411944:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12530.962701] LNet: Removed LNI 192.168.201.113@tcp [12531.900740] Key type .llcrypt unregistered [12531.903160] Key type ._llcrypt unregistered [12564.224157] Key type ._llcrypt registered [12564.227232] Key type .llcrypt registered [12564.346427] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [12564.739645] Key type .llcrypt unregistered [12564.743151] Key type ._llcrypt unregistered [12585.696976] Key type ._llcrypt registered [12585.701121] Key type .llcrypt registered [12585.799143] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [12586.976062] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12587.065303] alg: No test for adler32 (adler32-zlib) [12588.187547] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [12588.413803] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [12590.162317] Key type lgssc registered [12591.544538] Lustre: Echo OBD driver; http://www.lustre.org/ [12600.434186] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12600.466084] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12601.976765] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12608.364430] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [12612.578654] LustreError: 414165:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12612.586813] LustreError: 414165:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12613.615256] LustreError: 414164:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12618.732094] LustreError: 414165:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12620.563560] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12621.175816] LustreError: 414164:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12621.205183] LustreError: 414164:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12621.226867] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12626.405449] LustreError: 414165:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12626.429283] LustreError: 414165:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12628.481031] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12629.764592] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [12638.704064] LustreError: 414164:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12638.756055] LustreError: 414164:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [12642.345424] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12650.871951] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12651.364160] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12656.549643] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [12669.411115] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12669.418131] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12669.431197] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12671.968478] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12671.986155] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12671.998922] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12673.303856] Lustre: server umount lustre-OST0000 complete [12682.208435] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12682.211289] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12682.215313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12682.215321] Lustre: Skipped 1 previous similar message [12682.241602] Lustre: Skipped 1 previous similar message [12684.131802] Lustre: server umount lustre-MDT0000 complete [12687.337611] LustreError: 414161:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12687.355211] LustreError: 414161:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [12688.482943] LustreError: 414146:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453215 with bad export cookie 11721963388523084195 [12688.504614] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12689.050504] Lustre: server umount lustre-MDT0001 complete [12703.985604] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [12707.514632] Key type lgssc unregistered [12707.896660] LNet: 417320:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12707.900576] LNetError: 417320:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12707.925780] LNet: Removed LNI 192.168.201.113@tcp [12709.060126] Key type .llcrypt unregistered [12709.062954] Key type ._llcrypt unregistered [12739.841389] Key type ._llcrypt registered [12739.848032] Key type .llcrypt registered [12740.025867] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [12741.336307] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12741.432773] alg: No test for adler32 (adler32-zlib) [12742.497922] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [12742.748658] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [12744.399263] Key type lgssc registered [12745.799551] Lustre: Echo OBD driver; http://www.lustre.org/ [12753.651992] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12753.668749] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12769.247717] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12774.367331] LustreError: 418709:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.113@tcp: failed processing log, type 4: rc = -110 [12801.056199] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12807.811491] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [12816.662558] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12824.348919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12824.859842] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12824.932457] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12829.561371] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [12830.188429] LustreError: 418735:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12830.205584] LustreError: 418735:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12835.303114] LustreError: 418734:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12835.330654] LustreError: 418734:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12839.547657] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12839.897708] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12844.312849] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [12853.039753] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12860.129950] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12870.627652] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12870.631116] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12870.645777] Lustre: Skipped 1 previous similar message [12870.657986] Lustre: Skipped 1 previous similar message [12873.433921] Lustre: server umount lustre-OST0000 complete [12880.277400] Lustre: server umount lustre-MDT0000 complete [12880.869406] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12880.870503] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12880.885717] LustreError: 419751:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12880.924217] Lustre: Skipped 1 previous similar message [12884.503991] LustreError: 418717:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453411 with bad export cookie 13748211002661967279 [12884.510733] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12884.522327] LustreError: 418717:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12884.870217] Lustre: server umount lustre-MDT0001 complete [12898.397158] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [12901.644505] Key type lgssc unregistered [12902.012603] LNet: 422281:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12902.025034] LNetError: 422281:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12902.061649] LNet: Removed LNI 192.168.201.113@tcp [12903.154439] Key type .llcrypt unregistered [12903.159072] Key type ._llcrypt unregistered [12932.474822] Key type ._llcrypt registered [12932.477761] Key type .llcrypt registered [12932.605810] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [12934.281934] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12934.350858] alg: No test for adler32 (adler32-zlib) [12935.519272] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [12935.789310] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [12937.463733] Key type lgssc registered [12938.695872] Lustre: Echo OBD driver; http://www.lustre.org/ [12948.379985] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12948.394435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12949.876949] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12954.792605] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [12959.208510] LustreError: 423702:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12959.227047] LustreError: 423702:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12963.500673] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12963.881588] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12968.117873] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [12976.434786] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12982.893247] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12991.901383] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12992.367763] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12994.448857] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12999.911306] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [13010.730433] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13018.578968] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.13@tcp (stopping) [13019.096757] Lustre: server umount lustre-MDT0000 complete [13019.134488] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13019.167552] Lustre: Skipped 1 previous similar message [13019.168443] LustreError: 423702:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13019.207479] LustreError: 423702:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13022.558890] LustreError: 423682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.13@tcp arrived at 1788453549 with bad export cookie 5245846074245815188 [13022.587576] LustreError: 423682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13024.226427] LustreError: 425421:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13024.242321] LustreError: 425421:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13024.642784] LustreError: 424436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453551 with bad export cookie 5245846074245814292 [13024.661741] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13029.350487] LustreError: 423703:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13029.364398] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13029.364399] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13029.364406] Lustre: Skipped 1 previous similar message [13029.365044] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13029.384643] LustreError: 423703:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13030.966762] Lustre: server umount lustre-OST0000 complete [13034.465818] LustreError: 423702:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13034.509171] LustreError: 423702:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13040.674940] Lustre: server umount lustre-MDT0001 complete [13055.291227] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [13059.191849] Key type lgssc unregistered [13059.461648] LNet: 427392:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13059.469716] LNetError: 427392:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13059.492912] LNet: Removed LNI 192.168.201.113@tcp [13060.473586] Key type .llcrypt unregistered [13060.476524] Key type ._llcrypt unregistered [13093.351828] Key type ._llcrypt registered [13093.354688] Key type .llcrypt registered [13093.483759] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [13094.703484] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13094.989305] alg: No test for adler32 (adler32-zlib) [13096.026594] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [13096.348904] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [13098.079259] Key type lgssc registered [13099.193948] Lustre: Echo OBD driver; http://www.lustre.org/ [13107.588097] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13107.610576] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13109.214637] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13114.197129] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [13119.467931] LustreError: 428808:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13119.494726] LustreError: 428808:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13123.750632] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13124.218353] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13129.746830] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [13139.144415] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13146.113732] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13153.638341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13154.015670] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13155.062684] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [13162.517327] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [13173.246630] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13179.891376] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13179.908527] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13179.923399] Lustre: Skipped 1 previous similar message [13181.399561] LustreError: 428803:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13181.419160] Lustre: server umount lustre-MDT0001 complete [13184.992049] LustreError: 429536:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13185.024821] LustreError: 429536:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13190.126316] LustreError: 428809:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13190.128726] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13190.146977] Lustre: Skipped 1 previous similar message [13190.151718] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13195.239253] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13195.239435] LustreError: 428808:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13195.804863] Lustre: server umount lustre-OST0000 complete [13200.020891] Lustre: server umount lustre-MDT0000 complete [13216.705961] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [13220.263966] Key type lgssc unregistered [13220.616748] LNet: 432496:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13220.622719] LNetError: 432496:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13220.640142] LNet: Removed LNI 192.168.201.113@tcp [13221.556358] Key type .llcrypt unregistered [13221.558579] Key type ._llcrypt unregistered [13250.344688] Key type ._llcrypt registered [13250.349681] Key type .llcrypt registered [13250.495846] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [13267.774870] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [13268.826713] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13268.876302] alg: No test for adler32 (adler32-zlib) [13270.104349] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [13270.482274] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [13272.263183] Key type lgssc registered [13273.573029] Lustre: Echo OBD driver; http://www.lustre.org/ [13317.190326] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [13333.558147] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13333.589432] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13334.902700] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13334.925404] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13335.021342] Lustre: lustre-MDT0000: new disk, initializing [13335.131337] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13335.158168] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13339.314157] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [13356.978823] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13357.347804] Lustre: lustre-OST0000: new disk, initializing [13357.355869] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13357.366061] Lustre: Skipped 1 previous similar message [13357.381320] Lustre: 437535:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13357.548132] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13363.333369] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13363.356561] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13363.429945] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13366.059413] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [13375.625915] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13390.915952] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13391.036598] Lustre: 436517:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13391.124372] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13391.156800] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.13@tcp (not set up) [13391.235287] Lustre: lustre-MDT0001: new disk, initializing [13391.330330] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13391.364354] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13391.370854] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13395.703843] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [13405.548336] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13405.810199] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13411.394464] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13411.598900] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13420.011476] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13420.020169] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13420.023472] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13420.053330] Lustre: Skipped 1 previous similar message [13422.050953] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.13@tcp (stopping) [13422.061458] Lustre: Skipped 1 previous similar message [13423.757976] Lustre: server umount lustre-MDT0001 complete [13430.241093] LustreError: 436527:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13430.251242] LustreError: 436527:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [13435.377633] LustreError: 439230:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13435.402168] LustreError: 439230:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13440.490518] LustreError: 436523:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13445.601185] LustreError: 436528:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13445.610417] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13445.626976] LustreError: 436528:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13445.652667] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13445.675417] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13448.671322] Lustre: server umount lustre-OST0000 complete [13450.739752] LustreError: 436527:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13458.702872] Lustre: server umount lustre-MDT0000 complete [13473.047878] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [13475.945931] Key type lgssc unregistered [13476.295832] LNet: 441285:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13476.302738] LNetError: 441285:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13476.319435] LNet: Removed LNI 192.168.201.113@tcp [13477.441171] Key type .llcrypt unregistered [13477.444401] Key type ._llcrypt unregistered [13531.357221] Key type ._llcrypt registered [13531.366192] Key type .llcrypt registered [13531.536766] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [13531.902931] Key type .llcrypt unregistered [13531.908184] Key type ._llcrypt unregistered [13554.464504] Key type ._llcrypt registered [13554.466614] Key type .llcrypt registered [13554.563534] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [13555.510204] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13555.527877] alg: No test for adler32 (adler32-zlib) [13556.635378] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [13556.792463] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [13558.447190] Key type lgssc registered [13559.722422] Lustre: Echo OBD driver; http://www.lustre.org/ [13592.988235] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13593.050494] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13594.465341] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13594.541803] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13594.660677] Lustre: lustre-MDT0000: new disk, initializing [13594.798383] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13594.824536] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13600.867349] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [13613.905884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13614.085855] Lustre: 444671:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13614.144629] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13614.148205] Lustre: Skipped 1 previous similar message [13614.211417] Lustre: lustre-MDT0001: new disk, initializing [13614.331215] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13614.372113] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13614.390114] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13619.598413] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [13631.373203] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13637.560360] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13644.998792] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13645.286001] Lustre: lustre-OST0000: new disk, initializing [13645.289311] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13645.293655] Lustre: 446642:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13645.397226] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13652.615701] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [13655.096687] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13655.115560] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13655.197059] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13663.626487] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13676.008923] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13676.013623] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13676.025911] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13676.037362] Lustre: Skipped 2 previous similar messages [13676.055160] Lustre: Skipped 2 previous similar messages [13678.004256] Lustre: server umount lustre-MDT0000 complete [13683.459689] LustreError: 445467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788454210 with bad export cookie 18395213402870738988 [13683.467509] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13683.474849] LustreError: 445467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13684.141701] Lustre: server umount lustre-MDT0001 complete [13700.033101] Lustre: server umount lustre-OST0000 complete [13713.660486] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 12:50:39 (1788454239) [13723.201231] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13723.940066] LustreError: 448672:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13723.961401] LustreError: 448672:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [13724.106967] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13729.257271] LustreError: 448673:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13729.275674] LustreError: 448673:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13729.673385] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [13734.370694] LustreError: 448672:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13739.226937] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13739.515553] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13744.694233] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [13752.880532] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13759.003961] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13779.636896] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13780.050228] Lustre: Found index 0 for lustre-OST0000, updating log [13780.300971] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13787.283263] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [13795.800343] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13795.875831] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13805.028166] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13805.037809] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13805.059259] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13806.121401] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13806.141381] Lustre: Skipped 2 previous similar messages [13812.199102] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13812.213831] Lustre: Skipped 1 previous similar message [13817.316481] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13817.642595] Lustre: server umount lustre-OST0000 complete [13836.774220] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13836.783693] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13836.808194] Lustre: Skipped 1 previous similar message [13836.819322] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13838.827900] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13841.564399] Lustre: server umount lustre-MDT0000 complete [13842.977891] LustreError: 451179:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13842.987233] LustreError: 451179:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13845.665424] LNet: 1 local NIs in recovery (showing 1): 192.168.201.113@tcp [13846.691167] LustreError: 449408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788454373 with bad export cookie 18395213402870740381 [13846.692790] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13846.712165] LustreError: 449408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13847.160574] Lustre: server umount lustre-MDT0001 complete [13857.622237] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 12:53:03 (1788454383) [14486.848058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14487.527585] LustreError: 455101:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14487.557919] LustreError: 455101:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [14487.659937] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14492.486848] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [14492.640237] LustreError: 455102:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14497.765362] LustreError: 455101:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14503.907684] LustreError: 455102:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14503.925199] LustreError: 455102:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [14504.721733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14505.171697] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14510.842765] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [14521.513756] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14529.009454] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14548.255645] LNet: 1 local NIs in recovery (showing 1): 192.168.201.113@tcp [14551.370518] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14551.767173] Lustre: Found index 0 for lustre-OST0000, updating log [14552.101969] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14558.309511] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [14568.378882] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14572.072527] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14572.091462] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14572.099078] Lustre: Skipped 2 previous similar messages [14585.311397] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14585.717371] Lustre: server umount lustre-OST0000 complete [14592.485441] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14592.501835] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14592.535129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14594.597372] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14594.603241] Lustre: Skipped 4 previous similar messages [14598.047263] Lustre: server umount lustre-MDT0000 complete [14598.687910] LustreError: 455102:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14598.698594] LustreError: 455102:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [14602.662300] LustreError: 455081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788455129 with bad export cookie 18395213402870741466 [14602.670957] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14602.675531] LustreError: 455081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14603.309648] Lustre: server umount lustre-MDT0001 complete [14608.671300] LNet: 1 local NIs in recovery (showing 1): 192.168.201.113@tcp [14640.013138] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [14653.512896] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [14696.256347] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 13:07:02 (1788455222) [14719.387876] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [14722.867767] Key type lgssc unregistered [14723.275201] LNet: 461729:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14723.289410] LNetError: 461729:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14812.391219] LNet: Removed LNI 192.168.201.113@tcp [14813.459475] Key type .llcrypt unregistered [14813.463080] Key type ._llcrypt unregistered [14831.458856] Key type ._llcrypt registered [14831.461530] Key type .llcrypt registered [14831.576140] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [14832.888502] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14833.134517] alg: No test for adler32 (adler32-zlib) [14834.298924] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [14834.738530] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [14836.656718] Key type lgssc registered [14838.462788] Lustre: Echo OBD driver; http://www.lustre.org/ [14844.083584] LNet: Added LNI 192.168.201.113@tcp100 [8/256/0/180] [14845.145328] LNet: Added LNI 192.168.201.113@tcp101 [8/256/0/180] [14845.998804] LNet: Added LNI 192.168.201.113@tcp102 [8/256/0/180] [14846.819050] LNet: Added LNI 192.168.201.113@tcp103 [8/256/0/180] [14847.746654] LNet: Added LNI 192.168.201.113@tcp104 [8/256/0/180] [14848.690694] LNet: Added LNI 192.168.201.113@tcp105 [8/256/0/180] [14849.516860] LNet: Added LNI 192.168.201.113@tcp106 [8/256/0/180] [14850.593837] LNet: Added LNI 192.168.201.113@tcp107 [8/256/0/180] [14851.802179] LNet: Added LNI 192.168.201.113@tcp108 [8/256/0/180] [14853.066322] LNet: Added LNI 192.168.201.113@tcp109 [8/256/0/180] [14854.018296] LNet: Added LNI 192.168.201.113@tcp110 [8/256/0/180] [14855.115862] LNet: Added LNI 192.168.201.113@tcp111 [8/256/0/180] [14856.249795] LNet: Added LNI 192.168.201.113@tcp112 [8/256/0/180] [14857.451936] LNet: Added LNI 192.168.201.113@tcp113 [8/256/0/180] [14858.610673] LNet: Added LNI 192.168.201.113@tcp114 [8/256/0/180] [14859.385338] LNet: Added LNI 192.168.201.113@tcp115 [8/256/0/180] [14860.437667] LNet: Added LNI 192.168.201.113@tcp116 [8/256/0/180] [14861.518340] LNet: Added LNI 192.168.201.113@tcp117 [8/256/0/180] [14862.253678] LNet: Added LNI 192.168.201.113@tcp118 [8/256/0/180] [14863.153454] LNet: Added LNI 192.168.201.113@tcp119 [8/256/0/180] [14863.692211] LNet: Added LNI 192.168.201.113@tcp120 [8/256/0/180] [14864.601959] LNet: Added LNI 192.168.201.113@tcp121 [8/256/0/180] [14865.582929] LNet: Added LNI 192.168.201.113@tcp122 [8/256/0/180] [14866.512181] LNet: Added LNI 192.168.201.113@tcp123 [8/256/0/180] [14867.627430] LNet: Added LNI 192.168.201.113@tcp124 [8/256/0/180] [14868.687108] LNet: Added LNI 192.168.201.113@tcp125 [8/256/0/180] [14869.773594] LNet: Added LNI 192.168.201.113@tcp126 [8/256/0/180] [14870.788842] LNet: Added LNI 192.168.201.113@tcp127 [8/256/0/180] [14871.797841] LNet: Added LNI 192.168.201.113@tcp128 [8/256/0/180] [14872.595374] LNet: Added LNI 192.168.201.113@tcp129 [8/256/0/180] [14873.615772] LNet: Added LNI 192.168.201.113@tcp130 [8/256/0/180] [14874.562596] LNet: Added LNI 192.168.201.113@tcp131 [8/256/0/180] [14875.374029] LNet: Added LNI 192.168.201.113@tcp132 [8/256/0/180] [14876.252844] LNet: Added LNI 192.168.201.113@tcp133 [8/256/0/180] [14877.222136] LNet: Added LNI 192.168.201.113@tcp134 [8/256/0/180] [14878.125975] LNet: Added LNI 192.168.201.113@tcp135 [8/256/0/180] [14887.341153] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14887.368053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14888.527623] Lustre: 464760:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.113@tcp: can't add NID 192.168.201.113@tcp130: rc = -75 [14888.609154] LNetError: 462218:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.113 [14888.624322] LNetError: Protocol error connecting to 192.168.201.113@tcp117 at host 192.168.201.113:988: is it running a compatible version of Lustre? [14888.641218] LNetError: Skipped 3 previous similar messages [14888.759941] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14888.785486] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14888.850398] Lustre: lustre-MDT0000: new disk, initializing [14888.925883] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14888.940322] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14889.056988] LNet: 10 local NIs in recovery (showing 10): 192.168.201.113@tcp123, 192.168.201.113@tcp124, 192.168.201.113@tcp125, 192.168.201.113@tcp122, 192.168.201.113@tcp127, 192.168.201.113@tcp117, 192.168.201.113@tcp126, 192.168.201.113@tcp129, 192.168.201.113@tcp128, 192.168.201.113@tcp121 [14891.168453] LNetError: 462217:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.113 [14891.202173] LNetError: 462217:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 22 previous similar messages [14891.221285] LNetError: Protocol error connecting to 192.168.201.113@tcp123 at host 192.168.201.113:988: is it running a compatible version of Lustre? [14891.256932] LNetError: Skipped 18 previous similar messages [14895.282690] LNetError: 464789:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.113 [14895.317347] LNetError: 464789:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 2 previous similar messages [14895.338400] LNetError: Protocol error connecting to 192.168.201.113@tcp123 at host 192.168.201.113:988: is it running a compatible version of Lustre? [14897.496769] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [14903.456777] LNetError: 462215:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.113 [14903.472225] LNetError: Protocol error connecting to 192.168.201.113@tcp123 at host 192.168.201.113:988: is it running a compatible version of Lustre? [14913.414958] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14913.645708] Lustre: 464777:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14913.800375] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14913.808615] Lustre: Skipped 1 previous similar message [14913.903259] Lustre: lustre-MDT0001: new disk, initializing [14913.963527] Lustre: 465578:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.113@tcp: can't add NID 192.168.201.113@tcp130: rc = -75 [14913.985039] Lustre: 465578:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14914.311976] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14914.362501] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14914.393744] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14919.719586] LNetError: 462216:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.113 [14919.739512] LNetError: Protocol error connecting to 192.168.201.113@tcp118 at host 192.168.201.113:988: is it running a compatible version of Lustre? [14921.240945] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [14922.850899] Lustre: 465906:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.113@tcp: can't add NID 192.168.201.113@tcp130: rc = -75 [14922.862506] Lustre: 465906:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14927.911957] LNetError: 462217:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.113 [14927.934857] LNetError: 462217:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 6 previous similar messages [14927.962548] LNetError: Protocol error connecting to 192.168.201.113@tcp117 at host 192.168.201.113:988: is it running a compatible version of Lustre? [14928.030726] LNetError: Skipped 6 previous similar messages [14933.920414] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14941.179956] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14945.313386] LNetError: 462217:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.113 [14945.326691] LNetError: 462217:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 9 previous similar messages [14945.343826] LNetError: Protocol error connecting to 192.168.201.113@tcp118 at host 192.168.201.113:988: is it running a compatible version of Lustre? [14945.361517] LNetError: Skipped 9 previous similar messages [14949.268472] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14949.599191] LNet: 1 local NIs in recovery (showing 1): 192.168.201.113@tcp123 [14949.671878] Lustre: lustre-OST0000: new disk, initializing [14949.681135] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14949.696520] Lustre: 466752:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14949.823352] Lustre: 466757:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.113@tcp: can't add NID 192.168.201.113@tcp130: rc = -75 [14949.829942] Lustre: 466757:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14949.892802] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14958.158335] Lustre: 467052:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.113@tcp: can't add NID 192.168.201.113@tcp130: rc = -75 [14958.182319] Lustre: 467052:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14958.319066] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14958.332290] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14958.594816] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14959.111722] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [14972.336508] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14984.166892] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14984.167499] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14984.175785] LustreError: Skipped 1 previous similar message [14984.182052] Lustre: Skipped 1 previous similar message [14984.182595] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14984.292049] LNetError: 464786:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.113 [14984.307609] LNetError: 464786:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 15 previous similar messages [14984.327546] LNetError: Protocol error connecting to 192.168.201.113@tcp126 at host 192.168.201.113:988: is it running a compatible version of Lustre? [14984.341039] LNetError: Skipped 15 previous similar messages [14988.810892] Lustre: server umount lustre-OST0000 complete [14994.400588] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14994.416681] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14994.417085] Lustre: Skipped 1 previous similar message [14994.437949] Lustre: Skipped 2 previous similar messages [14999.574621] Lustre: server umount lustre-MDT0000 complete [15002.837370] LustreError: 465587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788455529 with bad export cookie 4219695611274955833 [15002.851920] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15003.320164] Lustre: server umount lustre-MDT0001 complete [15019.927579] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [15024.860413] Key type lgssc unregistered [15025.595785] LNet: 468671:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15025.618302] LNetError: 468671:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15025.634512] LNet: Removed LNI 192.168.201.113@tcp [15026.657995] LNet: Removed LNI 192.168.201.113@tcp100 [15026.663846] LNetError: 468671:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15026.700607] LNetError: 468671:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15026.725234] LNet: Removed LNI 192.168.201.113@tcp101 [15027.808415] LNet: Removed LNI 192.168.201.113@tcp102 [15027.817257] LNetError: 468671:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15027.828986] LNetError: 468671:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15027.843386] LNet: Removed LNI 192.168.201.113@tcp103 [15028.897531] LNet: Removed LNI 192.168.201.113@tcp104 [15029.919281] LNet: Removed LNI 192.168.201.113@tcp105 [15029.927125] LNetError: 468671:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15029.951097] LNetError: 468671:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [15029.968040] LNet: Removed LNI 192.168.201.113@tcp106 [15031.007128] LNet: Removed LNI 192.168.201.113@tcp107 [15032.031241] LNet: Removed LNI 192.168.201.113@tcp108 [15033.055349] LNet: Removed LNI 192.168.201.113@tcp109 [15034.081294] LNet: Removed LNI 192.168.201.113@tcp110 [15034.101738] LNetError: 468671:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15034.123792] LNetError: 468671:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [15034.136937] LNet: Removed LNI 192.168.201.113@tcp111 [15035.168025] LNet: Removed LNI 192.168.201.113@tcp112 [15036.198448] LNet: Removed LNI 192.168.201.113@tcp113 [15037.215658] LNet: Removed LNI 192.168.201.113@tcp114 [15038.240362] LNet: Removed LNI 192.168.201.113@tcp115 [15039.265549] LNet: Removed LNI 192.168.201.113@tcp116 [15040.287218] LNet: Removed LNI 192.168.201.113@tcp117 [15041.311214] LNet: Removed LNI 192.168.201.113@tcp118 [15042.336314] LNet: Removed LNI 192.168.201.113@tcp119 [15042.354236] LNetError: 468671:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15042.369738] LNetError: 468671:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [15042.389912] LNet: Removed LNI 192.168.201.113@tcp120 [15043.423247] LNet: Removed LNI 192.168.201.113@tcp121 [15044.447276] LNet: Removed LNI 192.168.201.113@tcp122 [15044.454168] LNet: Removed LNI 192.168.201.113@tcp123 [15045.538142] LNet: Removed LNI 192.168.201.113@tcp124 [15046.559160] LNet: Removed LNI 192.168.201.113@tcp125 [15047.583283] LNet: Removed LNI 192.168.201.113@tcp126 [15048.607283] LNet: Removed LNI 192.168.201.113@tcp127 [15049.631636] LNet: Removed LNI 192.168.201.113@tcp128 [15050.655739] LNet: Removed LNI 192.168.201.113@tcp129 [15050.659370] LNet: Removed LNI 192.168.201.113@tcp130 [15050.663970] LNet: Removed LNI 192.168.201.113@tcp131 [15050.666553] LNet: Removed LNI 192.168.201.113@tcp132 [15050.668588] LNet: Removed LNI 192.168.201.113@tcp133 [15050.670775] LNet: Removed LNI 192.168.201.113@tcp134 [15050.683321] LNet: Removed LNI 192.168.201.113@tcp135 [15051.817403] Key type .llcrypt unregistered [15051.819082] Key type ._llcrypt unregistered [15079.357532] Key type ._llcrypt registered [15079.359215] Key type .llcrypt registered [15079.445950] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [15095.280571] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [15097.693936] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15097.764973] alg: No test for adler32 (adler32-zlib) [15098.963632] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [15099.316401] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [15101.159667] Key type lgssc registered [15103.469046] Lustre: Echo OBD driver; http://www.lustre.org/ [15145.323706] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 13:14:31 (1788455671) [15165.693771] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [15168.602685] Key type lgssc unregistered [15168.904717] LNet: 472777:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15168.918927] LNetError: 472777:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15168.948437] LNet: Removed LNI 192.168.201.113@tcp [15169.909552] Key type .llcrypt unregistered [15169.911777] Key type ._llcrypt unregistered [15187.062932] Key type ._llcrypt registered [15187.066607] Key type .llcrypt registered [15187.145172] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [15188.175037] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15188.339044] alg: No test for adler32 (adler32-zlib) [15189.714099] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [15190.122725] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [15191.826404] Key type lgssc registered [15193.086862] Lustre: Echo OBD driver; http://www.lustre.org/ [15205.049156] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15205.091692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15206.576639] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15206.638399] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15206.751407] Lustre: lustre-MDT0000: new disk, initializing [15206.859051] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15206.888436] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15211.970340] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [15224.631588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15224.764524] Lustre: 473992:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15224.834698] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15224.838441] Lustre: Skipped 1 previous similar message [15224.899279] Lustre: lustre-MDT0001: new disk, initializing [15224.968252] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15225.020208] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15225.031949] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15229.720470] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [15239.669025] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15246.247337] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15253.990325] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15254.324443] Lustre: lustre-OST0000: new disk, initializing [15254.327467] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15254.333225] Lustre: 475962:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15254.433941] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15261.610674] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [15264.355927] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15264.374662] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15264.507647] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15273.686120] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15277.617170] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing lnet_if_list [15284.461851] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [15284.509948] LNet: Added LNI 192.168.201.113@tcp10 [8/256/0/180] [15290.830822] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [15290.861745] LNet: Added LNI 192.168.201.113@tcp11 [8/256/0/180] [15295.734589] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [15295.762633] LNet: Added LNI 192.168.201.113@tcp12 [8/256/0/180] [15300.933937] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [15300.965217] LNet: Added LNI 192.168.201.113@tcp13 [8/256/0/180] [15305.602224] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [15305.624958] LNet: Added LNI 192.168.201.113@tcp14 [8/256/0/180] [15310.638681] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [15310.663077] LNet: Added LNI 192.168.201.113@tcp15 [8/256/0/180] [15316.314239] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [15316.347586] LNet: Added LNI 192.168.201.113@tcp16 [8/256/0/180] [15321.339732] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [15321.363947] LNet: Added LNI 192.168.201.113@tcp17 [8/256/0/180] [15326.395602] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [15326.421448] LNet: Added LNI 192.168.201.113@tcp18 [8/256/0/180] [15331.662797] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [15331.694442] LNet: Added LNI 192.168.201.113@tcp19 [8/256/0/180] [15338.015539] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [15338.055892] LNet: Added LNI 192.168.201.113@tcp20 [8/256/0/180] [15343.585612] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [15343.641833] LNet: Added LNI 192.168.201.113@tcp21 [8/256/0/180] [15350.607277] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [15350.641660] LNet: Added LNI 192.168.201.113@tcp22 [8/256/0/180] [15358.111375] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [15358.138923] LNet: Added LNI 192.168.201.113@tcp23 [8/256/0/180] [15364.725478] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [15364.767970] LNet: Added LNI 192.168.201.113@tcp24 [8/256/0/180] [15370.393689] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [15370.422861] LNet: Added LNI 192.168.201.113@tcp25 [8/256/0/180] [15376.624386] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [15376.659827] LNet: Added LNI 192.168.201.113@tcp26 [8/256/0/180] [15383.152896] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [15383.179310] LNet: Added LNI 192.168.201.113@tcp27 [8/256/0/180] [15389.679425] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [15389.706663] LNet: Added LNI 192.168.201.113@tcp28 [8/256/0/180] [15396.310547] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [15396.341675] LNet: Added LNI 192.168.201.113@tcp29 [8/256/0/180] [15404.242713] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [15404.286638] LNet: Added LNI 192.168.201.113@tcp30 [8/256/0/180] [15412.637440] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [15412.667265] LNet: Added LNI 192.168.201.113@tcp31 [8/256/0/180] [15421.059725] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [15421.109145] LNet: Added LNI 192.168.201.113@tcp32 [8/256/0/180] [15428.407599] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [15428.434435] LNet: Added LNI 192.168.201.113@tcp33 [8/256/0/180] [15437.493465] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [15437.530419] LNet: Added LNI 192.168.201.113@tcp34 [8/256/0/180] [15445.108932] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [15445.142264] LNet: Added LNI 192.168.201.113@tcp35 [8/256/0/180] [15452.525366] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [15452.572807] LNet: Added LNI 192.168.201.113@tcp36 [8/256/0/180] [15461.934705] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [15462.012744] LNet: Added LNI 192.168.201.113@tcp37 [8/256/0/180] [15475.106537] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [15475.134171] LNet: Added LNI 192.168.201.113@tcp38 [8/256/0/180] [15483.695498] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [15483.738111] LNet: Added LNI 192.168.201.113@tcp39 [8/256/0/180] [15491.626673] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [15491.680254] LNet: Added LNI 192.168.201.113@tcp40 [8/256/0/180] [15500.972222] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [15501.000635] LNet: Added LNI 192.168.201.113@tcp41 [8/256/0/180] [15510.376217] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [15510.428600] LNet: Added LNI 192.168.201.113@tcp42 [8/256/0/180] [15521.921849] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [15521.942870] LNet: Added LNI 192.168.201.113@tcp43 [8/256/0/180] [15531.458799] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [15531.496933] LNet: Added LNI 192.168.201.113@tcp44 [8/256/0/180] [15543.004713] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [15543.048887] LNet: Added LNI 192.168.201.113@tcp45 [8/256/0/180] [15552.680265] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [15552.716974] LNet: Added LNI 192.168.201.113@tcp46 [8/256/0/180] [15562.051509] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [15562.094216] LNet: Added LNI 192.168.201.113@tcp47 [8/256/0/180] [15570.272403] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [15570.326851] LNet: Added LNI 192.168.201.113@tcp48 [8/256/0/180] [15578.995185] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [15579.026438] LNet: Added LNI 192.168.201.113@tcp49 [8/256/0/180] [15587.294521] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [15587.328179] LNet: Added LNI 192.168.201.113@tcp50 [8/256/0/180] [15607.784860] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15607.793552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15607.807257] Lustre: Skipped 2 previous similar messages [15607.825771] Lustre: Skipped 2 previous similar messages [15612.721660] Lustre: server umount lustre-MDT0000 complete [15617.160319] LustreError: 488995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788456144 with bad export cookie 10814979863012840009 [15617.199562] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15617.870227] Lustre: server umount lustre-MDT0001 complete [15634.399172] Lustre: 473266:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788456145/real 1788456145] req@00000000799f360f x1875331742092800/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788456161 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15634.447115] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15638.380789] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [15638.498911] Lustre: 473263:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788456149/real 1788456149] req@00000000d345766d x1875331742092928/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788456165 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15639.532056] Lustre: 473264:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788456150/real 1788456150] req@000000003df57494 x1875331742093056/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788456166 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15643.619422] Lustre: 473266:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788456154/real 1788456154] req@00000000d3c0894f x1875331742093312/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788456170 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15648.819155] Lustre: 473266:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788456159/real 1788456159] req@0000000030322fed x1875331742093440/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788456175 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15652.978429] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15662.239664] LustreError: 473262:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000768628e8 x1875331742094208/t0(0) o250->MGC192.168.201.113@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15662.553586] LustreError: 475956:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15662.568678] LustreError: 475956:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [15662.628454] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15663.651429] LustreError: 475957:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15663.656804] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15664.679153] LustreError: 475956:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15668.134234] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [15669.729126] LustreError: 475957:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15674.854722] LustreError: 475956:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15677.904401] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15678.373584] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15679.411103] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15683.868295] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [15692.613799] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15698.370451] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15724.000682] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15724.014801] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15724.042577] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15724.530026] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15725.539934] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15725.544197] Lustre: Skipped 2 previous similar messages [15727.460350] Lustre: server umount lustre-OST0000 complete [15734.756105] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15734.773130] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15734.799404] Lustre: Skipped 1 previous similar message [15734.818697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15738.780727] Lustre: server umount lustre-MDT0000 complete [15740.898279] LustreError: 500212:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15740.916164] LustreError: 500212:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [15743.154216] LustreError: 473985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788456270 with bad export cookie 10814979863012842599 [15743.162081] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15743.171110] LustreError: 473985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [15743.564180] Lustre: server umount lustre-MDT0001 complete [15755.682627] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [15758.594681] Key type lgssc unregistered [15759.297356] LNet: 503010:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15759.313879] LNetError: 503010:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15760.351845] LNet: Removed LNI 192.168.201.113@tcp [15760.359434] LNetError: 503010:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15760.372073] LNet: Removed LNI 192.168.201.113@tcp10 [15760.373698] LNet: Removed LNI 192.168.201.113@tcp11 [15760.381186] LNet: Removed LNI 192.168.201.113@tcp12 [15760.388527] LNet: Removed LNI 192.168.201.113@tcp13 [15760.393451] LNet: Removed LNI 192.168.201.113@tcp14 [15760.397646] LNet: Removed LNI 192.168.201.113@tcp15 [15760.409981] LNet: Removed LNI 192.168.201.113@tcp16 [15760.413572] LNet: Removed LNI 192.168.201.113@tcp17 [15760.419627] LNet: Removed LNI 192.168.201.113@tcp18 [15760.442407] LNet: Removed LNI 192.168.201.113@tcp19 [15760.449715] LNet: Removed LNI 192.168.201.113@tcp20 [15760.458725] LNet: Removed LNI 192.168.201.113@tcp21 [15760.468796] LNet: Removed LNI 192.168.201.113@tcp22 [15760.481921] LNet: Removed LNI 192.168.201.113@tcp23 [15760.487143] LNet: Removed LNI 192.168.201.113@tcp24 [15760.491176] LNet: Removed LNI 192.168.201.113@tcp25 [15760.497894] LNet: Removed LNI 192.168.201.113@tcp26 [15760.505978] LNet: Removed LNI 192.168.201.113@tcp27 [15760.510407] LNet: Removed LNI 192.168.201.113@tcp28 [15760.526307] LNet: Removed LNI 192.168.201.113@tcp29 [15760.528322] LNet: Removed LNI 192.168.201.113@tcp30 [15760.536879] LNet: Removed LNI 192.168.201.113@tcp31 [15760.552798] LNet: Removed LNI 192.168.201.113@tcp32 [15760.556258] LNet: Removed LNI 192.168.201.113@tcp33 [15760.566068] LNet: Removed LNI 192.168.201.113@tcp34 [15760.572960] LNet: Removed LNI 192.168.201.113@tcp35 [15760.575651] LNet: Removed LNI 192.168.201.113@tcp36 [15760.581810] LNet: Removed LNI 192.168.201.113@tcp37 [15760.595888] LNet: Removed LNI 192.168.201.113@tcp38 [15760.604995] LNet: Removed LNI 192.168.201.113@tcp39 [15760.609940] LNet: Removed LNI 192.168.201.113@tcp40 [15760.613132] LNet: Removed LNI 192.168.201.113@tcp41 [15760.621602] LNet: Removed LNI 192.168.201.113@tcp42 [15760.627711] LNet: Removed LNI 192.168.201.113@tcp43 [15760.633504] LNet: Removed LNI 192.168.201.113@tcp44 [15760.639906] LNet: Removed LNI 192.168.201.113@tcp45 [15760.659146] LNet: Removed LNI 192.168.201.113@tcp46 [15760.664487] LNet: Removed LNI 192.168.201.113@tcp47 [15760.670708] LNet: Removed LNI 192.168.201.113@tcp48 [15760.676675] LNet: Removed LNI 192.168.201.113@tcp49 [15760.692062] LNet: Removed LNI 192.168.201.113@tcp50 [15761.611673] Key type .llcrypt unregistered [15761.613861] Key type ._llcrypt unregistered [15787.230375] Key type ._llcrypt registered [15787.232445] Key type .llcrypt registered [15787.384614] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [15804.294888] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [15805.374862] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15805.511303] alg: No test for adler32 (adler32-zlib) [15806.586352] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [15806.801450] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [15808.479669] Key type lgssc registered [15809.560853] Lustre: Echo OBD driver; http://www.lustre.org/ [15852.506135] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15852.561801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15853.931352] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15853.967098] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15854.141526] Lustre: lustre-MDT0000: new disk, initializing [15854.335533] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15854.358295] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15861.192288] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [15876.361730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15876.502976] Lustre: 506217:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15876.683222] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15876.707210] Lustre: Skipped 1 previous similar message [15876.922560] Lustre: lustre-MDT0001: new disk, initializing [15877.103451] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15877.204719] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15877.228585] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15885.559033] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [15898.615851] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15906.012189] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15916.787937] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15917.066041] Lustre: *** cfs_fail_loc=90f, val=0*** [15917.179811] Lustre: lustre-OST0000: new disk, initializing [15917.187747] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15917.202478] Lustre: 508234:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15917.336412] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15925.450960] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15925.487737] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15925.742745] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15926.988994] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [15938.954278] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15956.449297] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15956.467059] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15956.471120] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15956.500713] Lustre: Skipped 1 previous similar message [15961.576078] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15961.587044] Lustre: Skipped 1 previous similar message [15966.175469] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15966.423964] Lustre: server umount lustre-OST0000 complete [15971.810057] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15971.830806] Lustre: Skipped 1 previous similar message [15971.833084] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15971.846669] Lustre: Skipped 2 previous similar messages [15977.034140] Lustre: server umount lustre-MDT0000 complete [15980.987434] LustreError: 506208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788456508 with bad export cookie 2819416429327231906 [15981.007566] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15981.604699] Lustre: server umount lustre-MDT0001 complete [15994.624447] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [15998.607702] Key type lgssc unregistered [15999.032840] LNet: 510144:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15999.037495] LNetError: 510144:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16000.113437] LNet: Removed LNI 192.168.201.113@tcp [16001.016529] Key type .llcrypt unregistered [16001.025872] Key type ._llcrypt unregistered [16022.706392] Key type ._llcrypt registered [16022.708658] Key type .llcrypt registered [16022.813342] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [16023.977517] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16024.159293] alg: No test for adler32 (adler32-zlib) [16025.292047] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [16025.559287] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [16027.343203] Key type lgssc registered [16028.492885] Lustre: Echo OBD driver; http://www.lustre.org/ [16037.844780] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16037.878604] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16039.559353] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16044.225803] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [16049.141817] LustreError: 511413:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16049.183689] LustreError: 511413:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [16054.245594] LustreError: 511414:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16055.856576] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16056.685569] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16063.026906] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [16075.000399] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16086.213923] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16094.754285] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16095.210443] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16103.452960] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [16113.719956] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16125.915215] loop0: detected capacity change from 0 to 204800000 [16126.191784] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16126.223694] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16126.243841] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16133.635334] loop0: detected capacity change from 0 to 204800000 [16133.974911] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16133.982589] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16133.996931] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16140.432915] loop0: detected capacity change from 0 to 204800000 [16147.260336] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16147.518826] Lustre: fs15246-OST0000: new disk, initializing [16147.526768] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [16147.545364] Lustre: 515010:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16147.643416] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16155.002106] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [16165.997942] loop1: detected capacity change from 0 to 204800000 [16173.683410] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16173.874046] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [16173.945632] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [16174.032567] Lustre: fs15246-MDT0000: new disk, initializing [16174.196224] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16174.218937] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16174.311657] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [16175.354087] Lustre: 516134:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@000000009d3794f4 x1875332617524992/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788456718 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [16179.572732] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [16184.501275] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [16184.517372] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [16184.575189] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [16214.501906] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16214.521377] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [16218.127949] Lustre: server umount fs15246-MDT0000 complete [16219.637557] LustreError: 511409:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16224.751553] LustreError: 511410:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16229.362480] Lustre: server umount fs15246-OST0000 complete [16244.717075] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16244.728267] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16244.743713] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16250.342020] LustreError: 513236:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16250.484657] Lustre: server umount lustre-OST0000 complete [16255.459629] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16255.478294] Lustre: Skipped 1 previous similar message [16255.485639] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16255.501258] Lustre: Skipped 3 previous similar messages [16260.581643] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16260.600627] Lustre: Skipped 1 previous similar message [16261.136777] Lustre: server umount lustre-MDT0000 complete [16265.704781] LustreError: 511414:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16265.733465] LustreError: 511414:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [16265.757408] LustreError: 511395:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788456792 with bad export cookie 13424457167341494975 [16265.768560] LustreError: 511395:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [16265.769595] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16266.248168] Lustre: server umount lustre-MDT0001 complete [16280.545679] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [16282.876956] Key type lgssc unregistered [16283.161983] LNet: 519238:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16283.167199] LNetError: 519238:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16283.184647] LNet: Removed LNI 192.168.201.113@tcp [16284.140056] Key type .llcrypt unregistered [16284.141495] Key type ._llcrypt unregistered [16309.827369] Key type ._llcrypt registered [16309.831777] Key type .llcrypt registered [16309.997780] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [16313.161163] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16313.203630] alg: No test for adler32 (adler32-zlib) [16314.487267] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [16314.839913] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [16316.679266] Key type lgssc registered [16318.387875] Lustre: Echo OBD driver; http://www.lustre.org/ [16366.754573] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [16383.026674] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [16424.510667] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16424.554112] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16425.998721] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16426.023934] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16426.092270] Lustre: lustre-MDT0000: new disk, initializing [16426.190734] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16426.215230] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16431.138982] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [16443.432459] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16443.701121] Lustre: 523730:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16443.807600] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16443.814767] Lustre: Skipped 1 previous similar message [16443.926281] Lustre: lustre-MDT0001: new disk, initializing [16444.052914] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16444.091610] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16444.111569] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16449.219083] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [16460.937647] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16468.159293] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16476.026365] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16476.485174] Lustre: lustre-OST0000: new disk, initializing [16476.498137] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16476.505303] Lustre: 525701:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16476.671875] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16483.968785] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16483.990888] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16484.264409] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16485.473296] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [16499.162768] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16504.884873] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [16505.202968] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [16511.514805] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [16511.820382] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [16515.059724] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16515.071408] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16515.073296] LustreError: Skipped 1 previous similar message [16515.098440] Lustre: Skipped 1 previous similar message [16515.102835] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16520.170706] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16520.176745] Lustre: Skipped 2 previous similar messages [16525.283399] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16525.291807] Lustre: Skipped 1 previous similar message [16528.352787] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16528.542822] Lustre: server umount lustre-OST0000 complete [16535.523197] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16535.543495] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16535.547470] Lustre: Skipped 1 previous similar message [16535.567699] Lustre: Skipped 1 previous similar message [16538.813756] Lustre: server umount lustre-MDT0000 complete [16542.995354] LustreError: 523722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788457070 with bad export cookie 7044972605453177203 [16542.999367] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16543.012903] LustreError: 523722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16543.390333] Lustre: server umount lustre-MDT0001 complete [16572.117972] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [16584.343300] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [16618.765578] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 16314 sec ========= 13:39:05 (1788457145) [16620.250487] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 13:39:06 (1788457146) === [16623.898915] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 13:39:10 (1788457150) === [16672.309853] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [16675.429799] Key type lgssc unregistered [16675.895561] LNet: 532353:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16675.905361] LNetError: 532353:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16675.941221] LNet: Removed LNI 192.168.201.113@tcp [16677.160823] Key type .llcrypt unregistered [16677.162530] Key type ._llcrypt unregistered