[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 580900917 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.996 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2448MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002392] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004015] kvm-guest: setup PV IPIs [ 0.007974] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229833f6470, max_idle_ns: 440795327230 ns [ 0.008023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399996) [ 0.009013] pid_max: default: 32768 minimum: 301 [ 0.010141] LSM: Security Framework initializing [ 0.011061] Yama: becoming mindful. [ 0.013026] SELinux: Initializing. [ 0.014072] *** VALIDATE selinux *** [ 0.024181] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.030446] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.031172] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033126] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.034121] *** VALIDATE tmpfs *** [ 0.036392] *** VALIDATE proc *** [ 0.037292] *** VALIDATE cgroup *** [ 0.038013] *** VALIDATE cgroup2 *** [ 0.039467] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.040268] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.042005] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.043029] Spectre V2 : User space: Vulnerable [ 0.044012] Speculative Store Bypass: Vulnerable [ 0.047629] debug: unmapping init [mem 0xffffffffb0059000-0xffffffffb0060fff] [ 0.049211] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.050728] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.051025] ... version: 2 [ 0.052015] ... bit width: 48 [ 0.053015] ... generic registers: 4 [ 0.054013] ... value mask: 0000ffffffffffff [ 0.055018] ... max period: 00007fffffffffff [ 0.056016] ... fixed-purpose events: 3 [ 0.057022] ... event mask: 000000070000000f [ 0.058292] rcu: Hierarchical SRCU implementation. [ 0.060789] smp: Bringing up secondary CPUs ... [ 0.061861] x86: Booting SMP configuration: [ 0.062024] .... node #0, CPUs: #1 #2 #3 [ 0.072185] smp: Brought up 1 node, 4 CPUs [ 0.074015] smpboot: Max logical packages: 1 [ 0.075020] smpboot: Total of 4 processors activated (19199.96 BogoMIPS) [ 0.154014] node 0 deferred pages initialised in 77ms [ 0.157073] devtmpfs: initialized [ 0.158475] x86/mm: Memory block size: 128MB [ 0.161523] gcov: version magic: 0x41383552 [ 0.164297] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.165101] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.166261] pinctrl core: initialized pinctrl subsystem [ 0.167209] [ 0.167783] ************************************************************* [ 0.168015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.169020] ** ** [ 0.170014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.171012] ** ** [ 0.172020] ** This means that this kernel is built to expose internal ** [ 0.173019] ** IOMMU data structures, which may compromise security on ** [ 0.174011] ** your system. ** [ 0.175012] ** ** [ 0.176012] ** If you see this message and you are not debugging the ** [ 0.177011] ** kernel, report this immediately to your vendor! ** [ 0.178011] ** ** [ 0.179009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.180014] ************************************************************* [ 0.181868] NET: Registered protocol family 16 [ 0.184644] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.187062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.190057] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.194009] cpuidle: using governor menu [ 0.195733] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.198553] PCI: Using configuration type 1 for base access [ 0.201125] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.212072] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.214105] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.218154] cryptd: max_cpu_qlen set to 1000 [ 0.223237] ACPI: Added _OSI(Module Device) [ 0.225013] ACPI: Added _OSI(Processor Device) [ 0.227013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.228011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.234722] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.242353] ACPI: Interpreter enabled [ 0.244068] ACPI: PM: (supports S0 S3 S4 S5) [ 0.245012] ACPI: Using IOAPIC for interrupt routing [ 0.247160] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.251489] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.264533] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.267044] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.271022] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.276082] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.283094] acpiphp: Slot [2] registered [ 0.287158] acpiphp: Slot [5] registered [ 0.289320] acpiphp: Slot [6] registered [ 0.291135] acpiphp: Slot [7] registered [ 0.292175] acpiphp: Slot [8] registered [ 0.294119] acpiphp: Slot [9] registered [ 0.296132] acpiphp: Slot [10] registered [ 0.297121] acpiphp: Slot [3] registered [ 0.299101] acpiphp: Slot [4] registered [ 0.301103] acpiphp: Slot [11] registered [ 0.303126] acpiphp: Slot [12] registered [ 0.305111] acpiphp: Slot [13] registered [ 0.307092] acpiphp: Slot [14] registered [ 0.309149] acpiphp: Slot [15] registered [ 0.312129] acpiphp: Slot [16] registered [ 0.315133] acpiphp: Slot [17] registered [ 0.317152] acpiphp: Slot [18] registered [ 0.319115] acpiphp: Slot [19] registered [ 0.322134] acpiphp: Slot [20] registered [ 0.325127] acpiphp: Slot [21] registered [ 0.327145] acpiphp: Slot [22] registered [ 0.330174] acpiphp: Slot [23] registered [ 0.332128] acpiphp: Slot [24] registered [ 0.333155] acpiphp: Slot [25] registered [ 0.335100] acpiphp: Slot [26] registered [ 0.337115] acpiphp: Slot [27] registered [ 0.339083] acpiphp: Slot [28] registered [ 0.341116] acpiphp: Slot [29] registered [ 0.343127] acpiphp: Slot [30] registered [ 0.345140] acpiphp: Slot [31] registered [ 0.346060] PCI host bridge to bus 0000:00 [ 0.348026] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.351023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.354027] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.356044] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.359027] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.363031] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.365180] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.369485] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.373636] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.386017] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.391086] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.394019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.397019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.401020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.405482] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.407649] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.410227] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.413000] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.416014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.429778] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.433745] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.438542] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.451017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.465015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.482021] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.495106] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.499017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.506016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.522028] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.536241] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.542017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.548019] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.563014] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.574791] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.582017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.589014] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.606020] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.625000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.631019] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.638018] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.655019] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.667046] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.674018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.683020] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.701018] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.715373] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.719538] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.722540] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.725585] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.728372] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.735087] iommu: Default domain type: Passthrough [ 0.736449] SCSI subsystem initialized [ 0.739154] ACPI: bus type USB registered [ 0.740103] usbcore: registered new interface driver usbfs [ 0.742091] usbcore: registered new interface driver hub [ 0.745093] usbcore: registered new device driver usb [ 0.747129] pps_core: LinuxPPS API ver. 1 registered [ 0.749010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.751080] PTP clock support registered [ 0.754295] EDAC MC: Ver: 3.0.0 [ 0.756087] PCI: Using ACPI for IRQ routing [ 0.758216] NetLabel: Initializing [ 0.760010] NetLabel: domain hash size = 128 [ 0.762011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.764259] NetLabel: unlabeled traffic allowed by default [ 0.767307] vgaarb: loaded [ 0.770454] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.773016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.780004] clocksource: Switched to clocksource kvm-clock [ 0.916444] VFS: Disk quotas dquot_6.6.0 [ 0.918562] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.921682] *** VALIDATE ramfs *** [ 0.923469] *** VALIDATE hugetlbfs *** [ 0.926198] pnp: PnP ACPI init [ 0.928814] pnp: PnP ACPI: found 6 devices [ 0.944966] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.950104] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.952902] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.956420] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.960483] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.963593] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.968068] NET: Registered protocol family 2 [ 0.970849] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.976424] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.980249] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.986067] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.990033] TCP: Hash tables configured (established 65536 bind 65536) [ 0.993693] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.997554] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.001479] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.007147] NET: Registered protocol family 1 [ 1.010336] RPC: Registered named UNIX socket transport module. [ 1.013580] RPC: Registered udp transport module. [ 1.015987] RPC: Registered tcp transport module. [ 1.018501] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.021612] NET: Registered protocol family 44 [ 1.024112] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.028209] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.031590] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.036174] PCI: CLS 0 bytes, default 64 [ 1.038605] Unpacking initramfs... [ 2.898637] debug: unmapping init [mem 0xffff9903bcc54000-0xffff9903bffbffff] [ 2.902256] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.904873] software IO TLB: mapped [mem 0x00000000b8c54000-0x00000000bcc54000] (64MB) [ 2.909784] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229833f6470, max_idle_ns: 440795327230 ns [ 3.617568] Initialise system trusted keyrings [ 3.619539] Key type blacklist registered [ 3.622158] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.638431] zbud: loaded [ 3.643920] *** VALIDATE nfs *** [ 3.645264] *** VALIDATE nfs4 *** [ 3.646824] pstore: using deflate compression [ 3.650512] Platform Keyring initialized [ 3.793167] NET: Registered protocol family 38 [ 3.795172] Key type asymmetric registered [ 3.797381] Asymmetric key parser 'x509' registered [ 3.800048] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.803857] io scheduler mq-deadline registered [ 3.805735] io scheduler kyber registered [ 3.807714] io scheduler bfq registered [ 3.809426] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.813507] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.818050] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.822229] ACPI: Power Button [PWRF] [ 3.828979] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.837280] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.852472] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.863743] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.892515] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.921527] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.951133] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.957287] Non-volatile memory driver v1.3 [ 3.959173] Linux agpgart interface v0.103 [ 4.006293] virtio_blk virtio1: [vda] 134584 512-byte logical blocks (68.9 MB/65.7 MiB) [ 4.011066] vda: detected capacity change from 0 to 68907008 [ 4.028654] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.032476] vdb: detected capacity change from 0 to 1073741824 [ 4.049678] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.053415] vdc: detected capacity change from 0 to 2621440000 [ 4.070464] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.074554] vdd: detected capacity change from 0 to 2621440000 [ 4.095366] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.099038] vde: detected capacity change from 0 to 4294967296 [ 4.118071] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.122943] vdf: detected capacity change from 0 to 4294967296 [ 4.134180] libphy: Fixed MDIO Bus: probed [ 4.141356] usbcore: registered new interface driver usbserial_generic [ 4.144798] usbserial: USB Serial support registered for generic [ 4.147574] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.152981] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.156191] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.159840] mousedev: PS/2 mouse device common for all mice [ 4.164080] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.164903] rtc_cmos 00:05: RTC can wake from S4 [ 4.180039] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.180251] rtc_cmos 00:05: registered as rtc0 [ 4.190924] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.194498] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.194554] intel_pstate: CPU model not supported [ 4.198342] hid: raw HID events driver (C) Jiri Kosina [ 4.207125] usbcore: registered new interface driver usbhid [ 4.210147] usbhid: USB HID core driver [ 4.212263] drop_monitor: Initializing network drop monitor service [ 4.215474] Initializing XFRM netlink socket [ 4.217959] NET: Registered protocol family 10 [ 4.221991] Segment Routing with IPv6 [ 4.223665] NET: Registered protocol family 17 [ 4.225729] mpls_gso: MPLS GSO support [ 4.233472] RAS: Correctable Errors collector initialized. [ 4.235573] AVX version of gcm_enc/dec engaged. [ 4.237637] AES CTR mode by8 optimization enabled [ 4.353638] sched_clock: Marking stable (4353603589, 0)->(5580569012, -1226965423) [ 4.358739] registered taskstats version 1 [ 4.361668] Loading compiled-in X.509 certificates [ 4.363497] zswap: loaded using pool lzo/zbud [ 4.398221] Key type big_key registered [ 4.415687] Key type encrypted registered [ 4.417684] ima: No TPM chip found, activating TPM-bypass! [ 4.420103] ima: Allocated hash algorithm: sha1 [ 4.422776] ima: No architecture policies found [ 4.425385] evm: Initialising EVM extended attributes: [ 4.428037] evm: security.selinux [ 4.429145] evm: security.ima [ 4.430307] evm: security.capability [ 4.431485] evm: HMAC attrs: 0x1 [ 4.434291] rtc_cmos 00:05: setting system clock to 2026-03-15 20:39:04 UTC (1773607144) [ 4.443547] debug: unmapping init [mem 0xffffffffb1003000-0xffffffffb11fffff] [ 4.446412] debug: unmapping init [mem 0xffffffffafd82000-0xffffffffb0058fff] [ 4.455660] Write protecting the kernel read-only data: 28672k [ 4.459501] debug: unmapping init [mem 0xffffffffae403000-0xffffffffae5fffff] [ 4.462931] debug: unmapping init [mem 0xffffffffaed14000-0xffffffffaedfffff] [ 4.509517] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.520606] systemd[1]: Detected virtualization kvm. [ 4.523103] systemd[1]: Detected architecture x86-64. [ 4.525456] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.556331] systemd[1]: No hostname configured. [ 4.558943] systemd[1]: Set hostname to . [ 4.561318] random: systemd: uninitialized urandom read (16 bytes read) [ 4.564151] systemd[1]: Initializing machine ID from random generator. [ 4.620068] random: ln: uninitialized urandom read (6 bytes read) [ 4.732474] random: systemd: uninitialized urandom read (16 bytes read) [ 4.736093] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 4.741231] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 4.746538] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Timers. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. Starting Journal Service... [ OK ] Reached target Sockets. Starting Apply Kernel Variables... Starting Setup Virtual Console... [ OK ] Reached target Local File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.543501] device-mapper: uevent: version 1.0.3 [ 5.546204] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 6.533351] virtio_net virtio0 ens2: renamed from eth0 [ 6.548597] random: fast init done [ 6.698368] scsi host0: ata_piix [ 6.796261] scsi host1: ata_piix [ 6.802962] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 6.805794] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.674406] dracut-initqueue[579]: RTNETLINK answers: File exists [ 10.938590] random: crng init done [ 10.940643] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 12.129594] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 13.597919] printk: systemd: 24 output lines suppressed due to ratelimiting [ 13.923926] SELinux: Disabled at runtime. [ 13.987384] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 13.997551] systemd[1]: Detected virtualization kvm. [ 14.000204] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 14.575443] systemd[1]: initrd-switch-root.service: Succeeded. [ 14.579993] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 14.590396] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 14.593706] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 14.597159] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 14.604297] systemd[1]: Starting Journal Service... Starting Journal Service... [ 14.613794] systemd[1]: Listening on RPCbind Server Activation Socket. [ OK ] Listening on RPCbind Server Activation Socket. Starting Apply Kernel Variables... [ OK ] Listening on initctl Compatibility Named Pipe. Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Control Socket. [ OK ] Stopped target Switch Root. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting udev Coldplug all Devices... [ OK ] Created slice system-getty.slice. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Initrd File Systems. Activating swap /dev/disk/by-label/SWAP... Mounting Huge Pages File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on Process Core Dump Socket. [ 14.737706] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Kernel Debug File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Initrd Root File System. Mounting POSIX Message Queue File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Journal Service. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 15.171073] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 15.515893] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 15.587798] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 15.707922] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 15.720385] EDAC sbridge: Ver: 1.1.2 [ 17.596548] Key type dns_resolver registered [ 18.068340] NFS: Registering the id_resolver key type [ 18.072092] Key type id_resolver registered [ 18.073903] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg105-server login: [ 64.016611] hrtimer: interrupt took 3213456 ns [ 102.952466] libcfs: loading out-of-tree module taints kernel. [ 103.055871] Key type ._llcrypt registered [ 103.070338] Key type .llcrypt registered [ 103.371502] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 140.096103] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 142.890876] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 142.909661] alg: No test for adler32 (adler32-zlib) [ 144.614605] Lustre: Lustre: Build Version: 2.17.51_1_gce01c51 [ 146.161675] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 148.248280] Key type lgssc registered [ 150.869451] Lustre: Echo OBD driver; http://www.lustre.org/ [ 178.770454] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 248.075439] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 270.324082] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 270.437752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 271.987816] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 272.075968] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 272.331878] Lustre: lustre-MDT0000: new disk, initializing [ 272.486707] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 272.510430] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 278.461794] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 297.736601] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 297.898876] Lustre: 6533:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 297.940593] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 297.946118] Lustre: Skipped 1 previous similar message [ 298.044447] Lustre: lustre-MDT0001: new disk, initializing [ 298.139060] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 298.176925] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 298.186046] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 304.935293] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 311.632082] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 327.478171] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 327.970349] Lustre: lustre-OST0000: new disk, initializing [ 327.976500] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 328.085390] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 333.868108] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 333.876874] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 333.941063] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 336.427966] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 357.477780] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 357.799990] Lustre: lustre-OST0001: new disk, initializing [ 357.830400] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 357.931049] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 365.150494] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 365.178208] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 365.270724] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 367.908660] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 384.248498] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 393.055129] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 408.299704] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing check_logdir /tmp/testlogs/ [ 416.935277] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing yml_node [ 423.105917] Lustre: DEBUG MARKER: Client: 2.17.51.1 [ 427.122021] Lustre: DEBUG MARKER: MDS: 2.17.51.1 [ 431.105655] Lustre: DEBUG MARKER: OSS: 2.17.51.1 [ 434.147974] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sun Mar 15 16:46:11 EDT 2026 [ 468.681962] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 472.064828] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 493.025313] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 493.044371] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 493.087671] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 493.574025] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 493.602754] Lustre: Skipped 1 previous similar message [ 493.610626] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 493.627534] Lustre: Skipped 2 previous similar messages [ 498.657448] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 498.679765] Lustre: Skipped 3 previous similar messages [ 503.777975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 503.790844] Lustre: Skipped 3 previous similar messages [ 507.361707] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 507.739700] Lustre: server umount lustre-MDT0000 complete [ 513.330797] LustreError: 6526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773607653 with bad export cookie 5249161844834540353 [ 513.334759] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 513.341360] LustreError: 6526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 514.027718] LustreError: 8886:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 514.030443] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 514.038099] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 514.066217] LustreError: 8886:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 6 previous similar messages [ 514.089623] Lustre: Skipped 2 previous similar messages [ 518.050839] LustreError: 6541:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 518.088579] LustreError: 6541:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 519.995649] Lustre: server umount lustre-MDT0001 complete [ 525.764716] Lustre: server umount lustre-OST0000 complete [ 531.672727] Lustre: server umount lustre-OST0001 complete [ 540.875993] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 556.812511] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 613.402295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 613.869900] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 614.024856] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 614.200201] Lustre: lustre-MDT0000: new disk, initializing [ 614.411248] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 614.477240] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 621.787653] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 638.597709] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 638.793961] Lustre: 15041:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 638.894731] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 638.905289] Lustre: Skipped 1 previous similar message [ 639.102360] Lustre: lustre-MDT0001: new disk, initializing [ 639.299246] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 639.384181] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 639.407471] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 646.512793] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 659.665318] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 668.566739] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 684.190378] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 684.841263] Lustre: lustre-OST0000: new disk, initializing [ 684.862266] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 685.081612] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 685.948378] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 685.965803] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 686.183311] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 695.358823] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 707.253991] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 714.122843] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 714.660390] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 721.212341] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 721.650562] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 727.018200] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 727.021781] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 727.039087] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 727.039098] Lustre: Skipped 3 previous similar messages [ 727.071226] Lustre: Skipped 1 previous similar message [ 731.995976] Lustre: server umount lustre-OST0000 complete [ 741.871629] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 741.912794] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 747.039564] Lustre: server umount lustre-MDT0000 complete [ 747.500626] LustreError: 15051:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 747.550693] LustreError: 15051:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 752.609965] LustreError: 15052:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 752.639578] LustreError: 15052:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 754.036774] LustreError: 16786:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773607894 with bad export cookie 5249161844834541879 [ 754.046200] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 754.060477] LustreError: 16786:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 754.829672] Lustre: server umount lustre-MDT0001 complete [ 761.372332] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 16:51:38 (1773607898) [ 765.695806] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 768.424465] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 16:51:45 (1773607905) [ 777.948942] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 778.602751] LustreError: 19056:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 778.800465] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 785.493640] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 787.943702] LustreError: 19057:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 787.980472] LustreError: 19057:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 797.963457] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 804.600976] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 813.522368] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 819.823149] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 831.312039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 841.849930] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 851.193607] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 867.543575] Lustre: server umount lustre-OST0000 complete [ 874.466109] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 874.478154] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 874.517509] Lustre: Skipped 2 previous similar messages [ 874.552830] Lustre: Skipped 5 previous similar messages [ 883.813631] Lustre: server umount lustre-MDT0000 complete [ 884.717182] LustreError: 20840:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 884.749666] LustreError: 20840:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 885.728469] Lustre: 3659:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773608009/real 1773608009] req@ffff990405cc2680 x1859762033733760/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1773608025 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'osp-pre-0-1.0' uid:0 gid:0 projid:4294967295 [ 885.778578] Lustre: 3659:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 889.479849] LustreError: 19036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773608029 with bad export cookie 5249161844834542768 [ 889.505581] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 889.517108] LustreError: 19036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 890.138709] Lustre: server umount lustre-MDT0001 complete [ 909.475810] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 914.208528] Key type lgssc unregistered [ 914.682767] LNet: 22285:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 914.696574] LNetError: 22285:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 915.757737] LNet: Removed LNI 192.168.201.105@tcp [ 917.400321] Key type .llcrypt unregistered [ 917.410327] Key type ._llcrypt unregistered [ 958.961120] Key type ._llcrypt registered [ 958.968067] Key type .llcrypt registered [ 959.179649] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 961.090152] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 961.390693] alg: No test for adler32 (adler32-zlib) [ 962.691985] Lustre: Lustre: Build Version: 2.17.51_1_gce01c51 [ 963.069487] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 964.824276] Key type lgssc registered [ 966.674958] Lustre: Echo OBD driver; http://www.lustre.org/ [ 980.595980] loop: module loaded [ 980.725041] loop0: detected capacity change from 0 to 204800000 [ 981.157819] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 981.188055] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 981.956213] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 992.493813] loop0: detected capacity change from 0 to 204800000 [ 992.862825] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 992.885462] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 992.907846] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 1001.689282] loop0: detected capacity change from 0 to 204800000 [ 1011.043210] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1011.134745] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1012.502317] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1012.615976] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1012.817789] Lustre: lustre-MDT0000: new disk, initializing [ 1013.026546] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1013.062829] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1021.191425] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1034.155705] loop1: detected capacity change from 0 to 204800000 [ 1041.838225] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1042.345760] Lustre: lustre-OST07c6: new disk, initializing [ 1042.351714] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 1042.376483] Lustre: Skipped 1 previous similar message [ 1042.485893] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 1050.230249] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 1050.238404] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 1050.273558] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 1051.511642] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1057.456754] Lustre: 24446:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 1057.474714] Lustre: 24446:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 1057.483421] Lustre: 24446:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 1057.489388] Lustre: 24446:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 1057.498282] Lustre: 24446:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 1057.505758] Lustre: 24446:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 1079.420473] Lustre: 25624:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 1079.427170] Lustre: 25624:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 1079.434837] Lustre: 25624:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 1079.451333] Lustre: 25624:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/4 [ 1079.474892] Lustre: 25624:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 1079.499221] Lustre: 25624:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1085.748934] Lustre: 25624:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 1085.762189] Lustre: 25624:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 1 previous similar message [ 1085.780900] Lustre: 25624:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 1085.800827] Lustre: 25624:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 1085.812461] Lustre: 25624:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 1085.826424] Lustre: 25624:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 1085.842925] Lustre: 25624:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 1085.867531] Lustre: 25624:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 1085.885793] Lustre: 25624:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 1085.910575] Lustre: 25624:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 1085.931081] Lustre: 25624:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1085.949632] Lustre: 25624:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 1089.369714] Lustre: 24448:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 1089.390773] Lustre: 24448:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 1 previous similar message [ 1089.409227] Lustre: 24448:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 1089.420847] Lustre: 24448:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 1089.431363] Lustre: 24448:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 1089.441308] Lustre: 24448:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 1089.452168] Lustre: 24448:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/0, punch: 0/0/0, quota 4/150/0 [ 1089.460398] Lustre: 24448:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 1089.473368] Lustre: 24448:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 1089.479953] Lustre: 24448:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 1089.491441] Lustre: 24448:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1089.497831] Lustre: 24448:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 1098.391259] Lustre: 25624:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 1098.408075] Lustre: 25624:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 3 previous similar messages [ 1098.420701] Lustre: 25624:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 1098.448102] Lustre: 25624:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 1098.473247] Lustre: 25624:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 1098.485255] Lustre: 25624:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 1098.495205] Lustre: 25624:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/2 [ 1098.503435] Lustre: 25624:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 1098.512008] Lustre: 25624:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 1098.519978] Lustre: 25624:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 1098.541461] Lustre: 25624:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1098.559962] Lustre: 25624:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 1111.500766] Lustre: 24446:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 1111.526477] Lustre: 24446:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 5 previous similar messages [ 1111.537580] Lustre: 24446:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 1111.549979] Lustre: 24446:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 1111.560479] Lustre: 24446:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 1111.568938] Lustre: 24446:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 1111.585504] Lustre: 24446:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/2 [ 1111.620180] Lustre: 24446:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 1111.636572] Lustre: 24446:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 1111.647109] Lustre: 24446:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 1111.669174] Lustre: 24446:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1111.692489] Lustre: 24446:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 1167.842474] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1167.854331] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1167.885442] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 1172.415634] Lustre: server umount lustre-OST07c6 complete [ 1186.553654] Lustre: server umount lustre-MDT0000 complete [ 1213.662688] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 1217.965882] Key type lgssc unregistered [ 1218.459569] LNet: 27999:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1218.470562] LNetError: 27999:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1218.488254] LNet: Removed LNI 192.168.201.105@tcp [ 1219.937316] Key type .llcrypt unregistered [ 1219.939291] Key type ._llcrypt unregistered [ 1249.260235] Key type ._llcrypt registered [ 1249.301124] Key type .llcrypt registered [ 1249.507930] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 1251.855315] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1251.896421] alg: No test for adler32 (adler32-zlib) [ 1253.316778] Lustre: Lustre: Build Version: 2.17.51_1_gce01c51 [ 1253.724014] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 1255.552186] Key type lgssc registered [ 1257.317673] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1271.847661] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1271.891977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1274.459380] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1282.253445] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1284.580709] LustreError: 29267:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1284.631175] LustreError: 29267:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 1289.709179] LustreError: 29266:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1294.819186] LustreError: 29267:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1297.471908] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1298.231368] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1305.390520] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1315.369675] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1321.775793] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1333.064641] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1333.718340] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1343.361785] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1343.992628] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 1352.287087] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1378.786697] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1378.809161] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1378.836245] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1379.813407] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1379.817450] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1379.841097] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1383.004447] Lustre: server umount lustre-OST0000 complete [ 1390.059731] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1390.085366] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1390.100394] Lustre: Skipped 2 previous similar messages [ 1395.171741] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1395.182199] Lustre: Skipped 1 previous similar message [ 1395.787738] Lustre: server umount lustre-MDT0000 complete [ 1400.290032] LustreError: 30735:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1400.311801] LustreError: 30735:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 1401.001897] LustreError: 29247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773608541 with bad export cookie 8784364175111073023 [ 1401.008870] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1401.026699] LustreError: 29247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1401.611373] Lustre: server umount lustre-MDT0001 complete [ 1417.352277] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 1420.952337] Key type lgssc unregistered [ 1421.353105] LNet: 32667:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1421.391757] LNetError: 32667:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1422.440494] LNet: Removed LNI 192.168.201.105@tcp [ 1423.728428] Key type .llcrypt unregistered [ 1423.735862] Key type ._llcrypt unregistered [ 1451.354459] Key type ._llcrypt registered [ 1451.360035] Key type .llcrypt registered [ 1451.546850] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 1453.152519] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1453.243351] alg: No test for adler32 (adler32-zlib) [ 1454.499978] Lustre: Lustre: Build Version: 2.17.51_1_gce01c51 [ 1454.757858] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 1456.533436] Key type lgssc registered [ 1458.717648] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1470.083980] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1470.119956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1471.854893] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1477.884242] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1481.190412] LustreError: 33924:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1481.222315] LustreError: 33924:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 1486.311928] LustreError: 33925:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1491.434587] LustreError: 33924:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1492.405101] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1493.097751] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1501.017943] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1510.860723] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1517.257393] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1528.737899] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1529.436488] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1540.375847] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1541.766461] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1549.319603] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1564.645955] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1564.663758] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1564.713588] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1567.721756] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1567.729490] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1567.743506] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1569.765536] Lustre: server umount lustre-OST0000 complete [ 1577.959865] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1577.982391] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1577.992570] Lustre: Skipped 1 previous similar message [ 1578.007036] Lustre: Skipped 2 previous similar messages [ 1582.640445] Lustre: server umount lustre-MDT0000 complete [ 1583.075885] LustreError: 36084:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1583.104663] LustreError: 36084:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 1588.201786] LustreError: 33924:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1588.262590] LustreError: 33924:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 1589.961608] LustreError: 33904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773608730 with bad export cookie 5561279094190782735 [ 1589.979716] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1589.994478] LustreError: 33904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1590.908084] Lustre: server umount lustre-MDT0001 complete [ 1612.455500] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 1616.712619] Key type lgssc unregistered [ 1617.108507] LNet: 37152:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1617.130306] LNetError: 37152:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1617.154195] LNet: Removed LNI 192.168.201.105@tcp [ 1618.423722] Key type .llcrypt unregistered [ 1618.428975] Key type ._llcrypt unregistered [ 1659.220965] Key type ._llcrypt registered [ 1659.225559] Key type .llcrypt registered [ 1659.562712] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 1661.645652] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1661.863642] alg: No test for adler32 (adler32-zlib) [ 1663.531347] Lustre: Lustre: Build Version: 2.17.51_1_gce01c51 [ 1663.947789] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 1665.898052] Key type lgssc registered [ 1667.989116] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1682.648629] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1682.714620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1684.873285] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1694.056032] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1694.192264] LustreError: 38557:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1694.236844] LustreError: 38557:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 1699.328767] LustreError: 38558:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1704.427415] LustreError: 38557:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1707.587396] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1708.461310] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1715.299722] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1726.179487] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1732.715717] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1746.275892] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1747.549222] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1754.688456] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1759.729844] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1769.554306] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1780.194926] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1780.197225] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1780.209862] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1780.256231] Lustre: Skipped 2 previous similar messages [ 1782.952205] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.5@tcp (stopping) [ 1782.959618] Lustre: Skipped 2 previous similar messages [ 1785.231448] Lustre: server umount lustre-MDT0000 complete [ 1785.325835] LustreError: 38558:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1785.370814] LustreError: 38558:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 1790.437441] LustreError: 39668:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1790.466470] LustreError: 39668:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 1790.540746] LustreError: 38537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773608930 with bad export cookie 5485683603343964101 [ 1790.552502] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1790.561493] LustreError: 38537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1793.193480] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.5@tcp (stopping) [ 1795.559879] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1795.586345] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1797.216494] Lustre: server umount lustre-MDT0001 complete [ 1816.247465] Lustre: server umount lustre-OST0000 complete [ 1847.133686] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 1851.579617] Key type lgssc unregistered [ 1852.127768] LNet: 42086:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1852.141721] LNetError: 42086:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1852.163373] LNet: Removed LNI 192.168.201.105@tcp [ 1853.096138] Key type .llcrypt unregistered [ 1853.099278] Key type ._llcrypt unregistered [ 1893.340557] Key type ._llcrypt registered [ 1893.344396] Key type .llcrypt registered [ 1893.479186] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 1894.954715] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1895.251856] alg: No test for adler32 (adler32-zlib) [ 1896.766654] Lustre: Lustre: Build Version: 2.17.51_1_gce01c51 [ 1897.257507] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 1899.107367] Key type lgssc registered [ 1900.860398] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1912.336450] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1912.406660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1914.164759] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1921.399366] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1924.583425] LustreError: 43491:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1924.624413] LustreError: 43491:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 1929.717511] LustreError: 43490:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1934.828332] LustreError: 43491:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1935.593496] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1936.417149] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1942.782884] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1951.879945] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1958.051345] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1968.557805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1969.179739] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1979.047717] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1984.648580] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1988.874921] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2000.358138] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2000.358503] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2000.371083] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2000.400061] Lustre: Skipped 1 previous similar message [ 2002.123694] Lustre: lustre-OST0000: Not available for connect from 192.168.201.5@tcp (stopping) [ 2003.805377] Lustre: server umount lustre-OST0000 complete [ 2025.972664] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2026.000970] Lustre: Skipped 1 previous similar message [ 2026.016625] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2029.029138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2029.048414] Lustre: Skipped 2 previous similar messages [ 2032.598548] Lustre: server umount lustre-MDT0000 complete [ 2034.147894] LustreError: 43491:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2034.180338] LustreError: 43491:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 2037.859357] LustreError: 43471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773609177 with bad export cookie 1970175513678670959 [ 2037.860547] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2037.913349] LustreError: 43471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2044.784932] Lustre: server umount lustre-MDT0001 complete [ 2065.405976] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 2070.379848] Key type lgssc unregistered [ 2070.874274] LNet: 46877:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2070.886919] LNetError: 46877:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2070.915080] LNet: Removed LNI 192.168.201.105@tcp [ 2072.058283] Key type .llcrypt unregistered [ 2072.062340] Key type ._llcrypt unregistered [ 2112.827185] Key type ._llcrypt registered [ 2112.830332] Key type .llcrypt registered [ 2113.041693] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 2115.357423] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2115.409322] alg: No test for adler32 (adler32-zlib) [ 2116.608436] Lustre: Lustre: Build Version: 2.17.51_1_gce01c51 [ 2116.983613] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 2118.754098] Key type lgssc registered [ 2120.649812] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2134.014210] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2134.034558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2135.720825] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2141.879288] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2145.259765] LustreError: 48285:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2145.310789] LustreError: 48285:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 2150.373056] LustreError: 48284:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2155.504648] LustreError: 48285:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2155.746099] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2156.245231] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2163.922047] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2173.810648] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2180.030839] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2191.291459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2192.003464] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2201.371771] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2205.318781] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 2210.649873] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2220.201341] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 2225.761662] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 2239.221560] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-03-1517h16m15s [ 2241.753622] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 2244.761105] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.5@tcp (stopping) [ 2245.816908] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.5@tcp (stopping) [ 2246.633072] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2246.659832] Lustre: Skipped 2 previous similar messages [ 2247.715324] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2247.735299] Lustre: Skipped 5 previous similar messages [ 2251.470229] Lustre: server umount lustre-MDT0000 complete [ 2251.893095] LustreError: 48279:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2257.671274] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 2258.039570] LustreError: 48281:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2258.058319] LustreError: 48281:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 2266.093619] LustreError: 48284:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2266.143934] LustreError: 48284:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 6 previous similar messages [ 2268.128243] Lustre: 47595:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773609392/real 1773609392] req@000000005a44f447 x1859764100314368/t0(0) o400->MGC192.168.201.105@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1773609408 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2268.203407] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2268.827848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2278.649844] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.5@tcp (not set up) [ 2279.150587] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2279.320031] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 2280.194925] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2289.222477] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2295.860812] Lustre: DEBUG MARKER: Wait for df (36273) ... [ 2298.484630] Lustre: DEBUG MARKER: done [ 2306.038038] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2306.044621] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2306.077467] Lustre: Skipped 1 previous similar message [ 2311.898418] Lustre: server umount lustre-OST0000 complete [ 2321.394858] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2321.406359] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2321.447075] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2321.477216] Lustre: Skipped 1 previous similar message [ 2321.488627] Lustre: Skipped 3 previous similar messages [ 2326.357719] Lustre: server umount lustre-MDT0000 complete [ 2329.065111] LustreError: 48284:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2329.085369] LustreError: 48284:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 9 previous similar messages [ 2333.146472] LustreError: 48263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773609473 with bad export cookie 7942318283624067168 [ 2333.148866] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2333.168481] LustreError: 48263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2333.818727] Lustre: server umount lustre-MDT0001 complete [ 2354.250772] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 2358.752668] Key type lgssc unregistered [ 2359.179883] LNet: 52875:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2359.188691] LNetError: 52875:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2359.200660] LNet: Removed LNI 192.168.201.105@tcp [ 2360.406755] Key type .llcrypt unregistered [ 2360.410055] Key type ._llcrypt unregistered [ 2432.207712] Key type ._llcrypt registered [ 2432.211056] Key type .llcrypt registered [ 2432.337529] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 2434.746038] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2434.795678] alg: No test for adler32 (adler32-zlib) [ 2436.020538] Lustre: Lustre: Build Version: 2.17.51_1_gce01c51 [ 2436.362225] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 2438.107961] Key type lgssc registered [ 2439.713367] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2452.525157] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2452.548497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2454.238735] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2454.306110] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2454.508433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2462.294084] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2479.823397] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2480.235695] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2480.346917] Lustre: 55073:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2480.707238] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2489.225149] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2499.594789] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2505.490337] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2517.170970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2517.611823] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2518.044969] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2525.865900] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 2528.149416] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2537.566375] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2548.006256] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-03-1517h21m24s [ 2550.815707] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 2561.080312] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 2565.091187] Lustre: DEBUG MARKER: Creating a test file and stat it [ 2565.145260] LustreError: 55078:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 2585.285309] Lustre: lustre-MDT0000: Client 993536e7-239a-4411-9693-76797b261055 (at 192.168.201.5@tcp) reconnecting [ 2605.224152] LustreError: 55078:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 2608.691417] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 2613.308597] Lustre: DEBUG MARKER: done [ 2628.069303] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2628.092638] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2628.120786] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2628.592622] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2628.610862] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2628.669796] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2629.882925] Lustre: server umount lustre-OST0000 complete [ 2638.817704] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2638.822952] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2638.839991] Lustre: Skipped 1 previous similar message [ 2641.717248] Lustre: server umount lustre-MDT0000 complete [ 2647.369197] LustreError: 55066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773609787 with bad export cookie 13778386851559314104 [ 2647.372759] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2647.388655] LustreError: 55066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2648.439979] Lustre: server umount lustre-MDT0001 complete [ 2665.836825] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 2670.376470] Key type lgssc unregistered [ 2670.861139] LNet: 59054:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2670.876419] LNetError: 59054:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2670.951740] LNet: Removed LNI 192.168.201.105@tcp [ 2672.439253] Key type .llcrypt unregistered [ 2672.442538] Key type ._llcrypt unregistered [ 2739.042710] Key type ._llcrypt registered [ 2739.045202] Key type .llcrypt registered [ 2739.262499] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 2741.267435] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2741.314866] alg: No test for adler32 (adler32-zlib) [ 2742.498780] Lustre: Lustre: Build Version: 2.17.51_1_gce01c51 [ 2742.773909] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 2744.600243] Key type lgssc registered [ 2746.783367] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2752.669597] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2752.692441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2753.999175] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2754.044143] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2754.271434] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2756.150558] Lustre: Failing over lustre-MDT0000 [ 2756.861852] Lustre: server umount lustre-MDT0000 complete [ 2770.466291] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 17:25:05 (1773609905) [ 2784.075534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2785.784195] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2792.198908] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2805.173483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2805.459634] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2805.567914] Lustre: 61846:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2806.232099] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2812.699678] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2820.842813] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2827.887346] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2836.973406] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2837.286397] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2837.551965] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2844.940565] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2846.978457] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2856.308621] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2863.786868] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2875.879895] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2875.885794] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2875.901631] Lustre: Skipped 2 previous similar messages [ 2875.925892] Lustre: Skipped 2 previous similar messages [ 2880.077042] Lustre: server umount lustre-MDT0000 complete [ 2884.714913] LustreError: 61840:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773610024 with bad export cookie 2046054986209348558 [ 2884.721150] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2884.727239] LustreError: 61840:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2885.215912] Lustre: server umount lustre-MDT0001 complete [ 2890.811433] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2896.876507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2902.496100] Lustre: 60765:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773610026/real 1773610026] req@00000000e8d3e337 x1859764755672576/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1773610042 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2902.533897] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2905.568660] Lustre: 60767:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773610029/real 1773610029] req@00000000f42f3828 x1859764755672704/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1773610045 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2907.618279] Lustre: 60768:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773610031/real 1773610031] req@000000000f165735 x1859764755672832/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1773610047 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2908.718103] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2909.752425] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x1c650c2b63ecd4b1 [ 2909.783135] Lustre: MGC192.168.201.105@tcp: Connection restored to 0@lo (at 0@lo) [ 2910.089665] LustreError: 63472:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2910.119345] LustreError: 63472:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 4 previous similar messages [ 2910.242723] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2910.430273] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2911.211426] Lustre: 60765:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773610034/real 1773610034] req@00000000617eb43d x1859764755673088/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1773610050 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2911.279865] LustreError: 63473:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2912.299161] LustreError: 65281:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2912.331104] LustreError: 65281:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 2915.339247] Lustre: 60765:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773610039/real 1773610039] req@00000000ae6c67f1 x1859764755673216/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1773610055 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2916.545992] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2917.349707] LustreError: 65284:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2917.386977] LustreError: 65284:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 2922.477075] LustreError: 65281:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2922.499566] LustreError: 65281:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 2928.065158] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2928.590952] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2931.701686] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2934.799337] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2944.730670] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2950.315327] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2962.406355] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2962.426542] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2962.434369] Lustre: Skipped 2 previous similar messages [ 2962.452759] Lustre: Skipped 2 previous similar messages [ 2966.540843] Lustre: server umount lustre-MDT0000 complete [ 2967.523060] LustreError: 65281:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2967.548766] LustreError: 65281:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 4 previous similar messages [ 2972.409240] LustreError: 61840:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773610112 with bad export cookie 2046054986209350833 [ 2972.414079] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2972.416982] LustreError: 61840:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2972.643149] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2972.873689] Lustre: server umount lustre-MDT0001 complete [ 2983.447060] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2999.161039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3018.239503] LustreError: 63473:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3018.262961] LustreError: 63473:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 3018.393401] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3018.517092] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 3025.917491] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3027.943324] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3027.969623] Lustre: Skipped 1 previous similar message [ 3039.315600] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3045.396599] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3046.510276] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3055.139741] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3062.042838] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3076.071226] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3076.071715] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3076.094422] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3076.105760] Lustre: Skipped 1 previous similar message [ 3078.434268] Lustre: server umount lustre-MDT0000 complete [ 3081.193290] LustreError: 68466:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3081.243418] LustreError: 68466:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 7 previous similar messages [ 3083.858531] LustreError: 61839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773610223 with bad export cookie 2046054986209352212 [ 3083.870091] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3083.894101] LustreError: 61839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3084.482910] Lustre: server umount lustre-MDT0001 complete [ 3093.842881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3099.520188] Lustre: DEBUG MARKER: files compared the same [ 3112.995198] Lustre: server umount lustre-OST0000 complete [ 3134.737179] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 3138.591528] Key type lgssc unregistered [ 3139.092948] LNet: 70788:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3139.111488] LNetError: 70788:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3140.134074] LNet: Removed LNI 192.168.201.105@tcp [ 3141.498441] Key type .llcrypt unregistered [ 3141.506260] Key type ._llcrypt unregistered [ 3176.405556] Key type ._llcrypt registered [ 3176.408671] Key type .llcrypt registered [ 3176.542428] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 3178.216933] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3178.266833] alg: No test for adler32 (adler32-zlib) [ 3179.511643] Lustre: Lustre: Build Version: 2.17.51_1_gce01c51 [ 3179.797775] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 3181.512182] Key type lgssc registered [ 3183.296994] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3193.960536] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3193.997509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3195.731346] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3201.846585] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug +malloc all [ 3205.097310] LustreError: 72189:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3205.140511] LustreError: 72189:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 3210.217469] LustreError: 72190:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3215.330123] LustreError: 72189:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3215.528719] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3216.165219] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3222.053755] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug +malloc all [ 3230.719830] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3237.262224] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3246.111484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3246.715456] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3254.431695] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug +malloc all [ 3261.737053] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3265.140873] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 3280.877449] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3280.878668] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3280.898136] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3280.900865] Lustre: Skipped 1 previous similar message [ 3284.568135] Lustre: server umount lustre-OST0000 complete [ 3291.106515] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3291.133824] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3291.149831] Lustre: Skipped 2 previous similar messages [ 3295.854765] Lustre: server umount lustre-MDT0000 complete [ 3296.233781] LustreError: 72903:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3296.266585] LustreError: 72903:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 3301.346877] LustreError: 73352:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3301.386219] LustreError: 73352:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 3301.579832] LustreError: 72171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773610441 with bad export cookie 9788944726787203615 [ 3301.584820] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3301.593642] LustreError: 72171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3302.357665] Lustre: server umount lustre-MDT0001 complete [ 3316.766989] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 3321.739811] Key type lgssc unregistered [ 3322.091761] LNet: 75400:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3322.105220] LNetError: 75400:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3322.137561] LNet: Removed LNI 192.168.201.105@tcp [ 3323.731379] Key type .llcrypt unregistered [ 3323.741974] Key type ._llcrypt unregistered [ 3363.248889] Key type ._llcrypt registered [ 3363.254225] Key type .llcrypt registered [ 3363.407941] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 3364.949765] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3365.042266] alg: No test for adler32 (adler32-zlib) [ 3366.330270] Lustre: Lustre: Build Version: 2.17.51_1_gce01c51 [ 3366.572605] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 3368.248862] Key type lgssc registered [ 3370.101341] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3381.287496] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3381.313197] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3396.897178] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3402.016585] LustreError: 76777:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.105@tcp: failed processing log, type 4: rc = -110 [ 3428.705859] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3438.808542] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3448.620837] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3460.254295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3460.546356] Lustre: *** cfs_fail_loc=706, val=0*** [ 3461.221191] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3461.442256] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 3468.252513] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3468.782108] LustreError: 77752:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3468.814509] LustreError: 77752:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 5 previous similar messages [ 3473.894171] LustreError: 77751:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3473.924108] LustreError: 77751:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 3479.016091] LustreError: 78039:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3479.033584] LustreError: 78039:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 3481.008152] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3481.388008] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3487.023368] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3496.257354] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3501.407976] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3507.199425] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3507.203021] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3507.217293] LustreError: Skipped 1 previous similar message [ 3507.220716] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3507.265579] Lustre: Skipped 1 previous similar message [ 3511.221525] Lustre: server umount lustre-OST0000 complete [ 3517.409853] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3517.416837] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3517.445773] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3517.475470] Lustre: Skipped 2 previous similar messages [ 3522.534121] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3522.544631] Lustre: Skipped 1 previous similar message [ 3525.038186] Lustre: server umount lustre-MDT0000 complete [ 3527.658436] LustreError: 77750:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3527.691150] LustreError: 77750:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 3530.252863] LustreError: 76783:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773610670 with bad export cookie 3191549336560141719 [ 3530.257405] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3530.265496] LustreError: 76783:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3530.697637] Lustre: server umount lustre-MDT0001 complete [ 3545.296977] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 3549.199681] Key type lgssc unregistered [ 3549.677490] LNet: 80047:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3549.688619] LNetError: 80047:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3549.712988] LNet: Removed LNI 192.168.201.105@tcp [ 3550.711206] Key type .llcrypt unregistered [ 3550.713387] Key type ._llcrypt unregistered [ 3582.875649] Key type ._llcrypt registered [ 3582.878182] Key type .llcrypt registered [ 3583.054398] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 3584.087250] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3584.247020] alg: No test for adler32 (adler32-zlib) [ 3585.487270] Lustre: Lustre: Build Version: 2.17.51_1_gce01c51 [ 3585.929822] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 3587.672204] Key type lgssc registered [ 3589.400501] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3599.536933] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3599.560836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3606.527106] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3616.991966] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3617.795468] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3622.882602] LustreError: 82121:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3622.914041] LustreError: 82121:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 3625.071499] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3626.995180] LustreError: 82124:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3627.022523] LustreError: 82124:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 3632.099573] LustreError: 82125:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3632.120283] LustreError: 82125:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 3636.666447] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3637.235560] LustreError: 82124:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3637.269178] LustreError: 82124:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 3637.398375] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3642.533527] LustreError: 82125:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3642.552061] LustreError: 82125:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 3646.038744] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3653.089907] LustreError: 82119:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3653.133552] LustreError: 82119:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 5 previous similar messages [ 3658.134063] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3658.153272] Lustre: MGS: Client dd373ff4-e3ef-4548-b0b5-6b5fea01f673 (at 0@lo) reconnecting [ 3658.198782] LustreError: 82135:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 3658.224029] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [ 3658.233845] Lustre: Skipped 2 previous similar messages [ 3658.504251] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3658.608086] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 3663.839470] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3684.323043] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3684.331466] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3684.356297] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3684.397393] Lustre: Skipped 1 previous similar message [ 3687.956207] Lustre: server umount lustre-OST0000 complete [ 3694.561620] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3694.567230] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3694.590204] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3694.620315] Lustre: Skipped 2 previous similar messages [ 3699.684389] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3699.699467] Lustre: Skipped 1 previous similar message [ 3704.801416] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3704.814104] Lustre: Skipped 1 previous similar message [ 3708.384406] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3708.778381] Lustre: server umount lustre-MDT0000 complete [ 3709.925324] LustreError: 82124:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3709.945848] LustreError: 82124:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 3715.115143] Lustre: server umount lustre-MDT0001 complete [ 3720.708476] Lustre: server umount MGS complete [ 3738.442951] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 17:41:15 (1773610875) [ 3784.133070] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 3801.124372] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 3856.429571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3863.046721] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3877.274448] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3877.767441] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 3877.869820] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3878.020751] Lustre: lustre-MDT0001: new disk, initializing [ 3878.154404] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3878.211681] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3878.229520] Lustre: 89246:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 3878.252621] LustreError: 89246:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 3884.273682] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3899.171009] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3899.510592] Lustre: lustre-OST0000: new disk, initializing [ 3899.519642] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3899.657037] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3908.431882] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3909.234635] Lustre: 90303:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@000000009d447454 x1859765639622656/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1773611065 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3917.678818] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3926.632458] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3926.644137] Lustre: MGS: Client 1059542a-de0a-4fc5-864e-7d813a618ed4 (at 0@lo) reconnecting [ 3926.686477] Lustre: 88531:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3926.702514] LustreError: 89232:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 3926.718508] LustreError: 89232:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 3926.745425] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3926.829692] Lustre: lustre-MDT0000: new disk, initializing [ 3927.035328] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3927.082056] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3927.106663] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3928.163066] Lustre: 90957:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@000000002c0b0f7c x1859765639630208/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1773611084 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3932.820939] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3933.410455] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3936.757490] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 3936.779935] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 3937.333684] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 3937.505587] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 3963.378831] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3963.387543] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3963.418531] Lustre: Skipped 1 previous similar message [ 3963.433505] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3966.440394] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3969.386477] Lustre: server umount lustre-OST0000 complete [ 3976.675984] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3976.679973] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3976.732312] Lustre: Skipped 1 previous similar message [ 3980.290421] Lustre: server umount lustre-MDT0000 complete [ 3981.799644] LustreError: 91375:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3981.832347] LustreError: 91375:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 3993.270464] Lustre: server umount lustre-MDT0001 complete [ 3998.391492] Lustre: server umount MGS complete [ 4014.641391] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 17:45:52 (1773611152) [ 4040.603555] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 4044.780072] Key type lgssc unregistered [ 4045.253738] LNet: 93425:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4045.268622] LNetError: 93425:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4045.300513] LNet: Removed LNI 192.168.201.105@tcp [ 4046.444687] Key type .llcrypt unregistered [ 4046.448482] Key type ._llcrypt unregistered [ 4071.644795] Key type ._llcrypt registered [ 4071.647811] Key type .llcrypt registered [ 4071.791578] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 4073.269273] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4073.336239] alg: No test for adler32 (adler32-zlib) [ 4075.182989] Lustre: Lustre: Build Version: 2.17.51_1_gce01c51 [ 4075.469750] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 4077.202107] Key type lgssc registered [ 4079.357164] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4091.801914] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4091.854967] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4093.680633] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4100.098989] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4103.145680] LustreError: 94631:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4103.210072] LustreError: 94631:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 4108.259875] LustreError: 94632:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4113.367795] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4113.380862] LustreError: 94631:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4113.944585] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4120.470496] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4129.569549] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4135.458871] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4145.368419] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4145.841117] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4155.726242] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4163.268401] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 4164.982813] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4178.923436] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4178.926489] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4178.953304] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4178.966137] Lustre: Skipped 1 previous similar message [ 4179.015760] Lustre: Skipped 1 previous similar message [ 4183.762192] Lustre: server umount lustre-OST0000 complete [ 4189.170239] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4189.224142] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4189.235527] Lustre: Skipped 1 previous similar message [ 4194.273988] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4194.285189] Lustre: Skipped 1 previous similar message [ 4195.832927] Lustre: server umount lustre-MDT0000 complete [ 4199.399300] LustreError: 94632:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4199.445679] LustreError: 94632:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 4201.233347] LustreError: 94613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773611341 with bad export cookie 8290311526668783216 [ 4201.251969] LustreError: 94613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4201.254375] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4201.795513] Lustre: server umount lustre-MDT0001 complete [ 4217.099051] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 4221.156799] Key type lgssc unregistered [ 4221.609712] LNet: 97822:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4221.625585] LNetError: 97822:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4221.651243] LNet: Removed LNI 192.168.201.105@tcp [ 4222.941786] Key type .llcrypt unregistered [ 4222.947849] Key type ._llcrypt unregistered [ 4247.767859] Key type ._llcrypt registered [ 4247.769877] Key type .llcrypt registered [ 4247.928206] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 4249.745890] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4249.844407] alg: No test for adler32 (adler32-zlib) [ 4251.677797] Lustre: Lustre: Build Version: 2.17.51_1_gce01c51 [ 4252.116896] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 4253.934801] Key type lgssc registered [ 4256.656732] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4266.343213] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4266.351845] LustreError: 98912:0:(tgt_mount.c:2522:server_fill_super()) cfs_race id 716 sleeping [ 4266.388802] LustreError: 98916:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 waking [ 4266.394426] LustreError: 98912:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 awake: rc=4965 [ 4266.401303] LustreError: 98912:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 4266.416621] LustreError: 98912:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 4266.426315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4268.081112] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4278.248644] LustreError: 98954:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4278.278296] LustreError: 98954:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 4280.397498] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4280.956331] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4287.789873] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4297.312767] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing lsmod [ 4302.222620] LustreError: 100082:0:(tgt_mount.c:2522:server_fill_super()) cfs_race id 716 sleeping [ 4302.285697] LustreError: 100089:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 waking [ 4302.303658] LustreError: 100082:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 awake: rc=4941 [ 4302.314452] LustreError: 100082:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 4302.335071] LustreError: 100082:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 4302.352271] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4303.317624] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4310.504049] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4310.505980] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4310.552509] Lustre: Skipped 1 previous similar message [ 4310.606110] Lustre: Skipped 2 previous similar messages [ 4315.147428] Lustre: server umount lustre-MDT0000 complete [ 4315.634225] LustreError: 98950:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4315.675834] LustreError: 98950:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 4320.744814] LustreError: 98953:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4320.774645] LustreError: 98953:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 4321.530335] LustreError: 98931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773611461 with bad export cookie 7334972777488993970 [ 4321.546836] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4321.561838] LustreError: 98931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4322.075443] Lustre: server umount lustre-MDT0001 complete [ 4337.629782] Lustre: server umount lustre-OST0000 complete [ 4350.407698] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4351.325658] LustreError: 101160:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4351.538220] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4356.580027] LustreError: 101161:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4358.545233] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4366.817585] LustreError: 101160:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4366.851560] LustreError: 101160:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 4372.927427] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4373.739863] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4381.225944] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4390.417987] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4397.165446] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4408.014629] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4408.714054] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4418.903713] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4420.989224] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 4429.350184] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4441.578014] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4441.578093] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4441.595433] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4441.622973] Lustre: Skipped 2 previous similar messages [ 4441.642374] Lustre: Skipped 1 previous similar message [ 4446.691333] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4446.710337] Lustre: Skipped 1 previous similar message [ 4451.809989] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4451.826275] Lustre: Skipped 1 previous similar message [ 4455.392412] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4455.936841] Lustre: server umount lustre-OST0000 complete [ 4465.641633] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4465.663089] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4465.677817] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4477.409056] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4477.412235] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4477.451079] Lustre: Skipped 5 previous similar messages [ 4477.904263] Lustre: server umount lustre-MDT0000 complete [ 4482.536285] LustreError: 101161:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4482.565638] LustreError: 101161:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 4484.094045] LustreError: 101140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773611624 with bad export cookie 7334972777488994670 [ 4484.108806] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4484.120800] LustreError: 101140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4484.683428] Lustre: server umount lustre-MDT0001 complete [ 4504.785312] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 4509.787587] Key type lgssc unregistered [ 4510.272132] LNet: 104370:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4510.289767] LNetError: 104370:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4510.315243] LNet: Removed LNI 192.168.201.105@tcp [ 4511.899401] Key type .llcrypt unregistered [ 4511.904246] Key type ._llcrypt unregistered [ 4549.893875] Key type ._llcrypt registered [ 4549.905071] Key type .llcrypt registered [ 4550.027974] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 4552.347206] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4552.400475] alg: No test for adler32 (adler32-zlib) [ 4553.752767] Lustre: Lustre: Build Version: 2.17.51_1_gce01c51 [ 4554.171230] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 4555.936392] Key type lgssc registered [ 4557.711266] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4569.988775] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4570.029254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4571.746533] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4578.871712] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4581.867528] LustreError: 105774:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4581.903840] LustreError: 105774:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 4586.977741] LustreError: 105773:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4592.048315] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4592.104079] LustreError: 105774:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4592.631951] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4600.274217] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4608.967766] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4615.973070] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4627.059374] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4627.536295] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4637.315828] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4641.946540] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:67 to 0x240000401:97) [ 4648.580042] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4656.975868] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 4662.947346] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 4668.909112] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4668.920249] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4668.941889] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4672.996586] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4673.004831] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4673.048244] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4673.127281] Lustre: Skipped 1 previous similar message [ 4675.464308] Lustre: server umount lustre-OST0000 complete [ 4681.189477] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4681.193908] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4681.240784] Lustre: Skipped 1 previous similar message [ 4681.270228] Lustre: Skipped 1 previous similar message [ 4686.308864] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4686.326690] Lustre: Skipped 1 previous similar message [ 4687.519874] Lustre: server umount lustre-MDT0000 complete [ 4691.431857] LustreError: 105768:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4691.460311] LustreError: 105768:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 4692.683836] LustreError: 105755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773611832 with bad export cookie 13363566719424054323 [ 4692.693980] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4692.696031] LustreError: 105755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4693.033725] Lustre: server umount lustre-MDT0001 complete [ 4709.172123] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 4713.047859] Key type lgssc unregistered [ 4713.424421] LNet: 109065:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4713.436278] LNetError: 109065:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4713.456825] LNet: Removed LNI 192.168.201.105@tcp [ 4714.437923] Key type .llcrypt unregistered [ 4714.440532] Key type ._llcrypt unregistered [ 4740.594588] Key type ._llcrypt registered [ 4740.597500] Key type .llcrypt registered [ 4740.766712] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 4742.578611] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4742.751938] alg: No test for adler32 (adler32-zlib) [ 4744.104813] Lustre: Lustre: Build Version: 2.17.51_1_gce01c51 [ 4744.434901] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 4746.176650] Key type lgssc registered [ 4747.652942] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4758.773894] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4758.789971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4760.569945] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4767.201890] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4770.798030] LustreError: 110278:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4770.839310] LustreError: 110278:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 4775.921882] LustreError: 110277:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4780.946385] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4781.033569] LustreError: 110278:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4781.763238] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4788.985600] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4798.111354] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4803.516673] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4814.640049] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4815.164634] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4823.270697] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4830.607611] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 4831.115485] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4841.954951] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4841.971559] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4842.001555] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4846.058407] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4846.059151] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4846.059639] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4846.072203] Lustre: Skipped 1 previous similar message [ 4847.489230] Lustre: server umount lustre-OST0000 complete [ 4853.241709] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4853.260224] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4853.298886] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4856.290060] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4856.293194] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4856.323865] Lustre: Skipped 1 previous similar message [ 4858.784994] Lustre: server umount lustre-MDT0000 complete [ 4861.411182] LustreError: 111129:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4861.441751] LustreError: 111129:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 4864.211620] LustreError: 110995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773612004 with bad export cookie 1679677741147421881 [ 4864.228893] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4864.231670] LustreError: 110995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4865.003949] Lustre: server umount lustre-MDT0001 complete [ 4881.231207] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 4885.101860] Key type lgssc unregistered [ 4885.413607] LNet: 113461:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4885.420143] LNetError: 113461:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4885.440257] LNet: Removed LNI 192.168.201.105@tcp [ 4886.305851] Key type .llcrypt unregistered [ 4886.310123] Key type ._llcrypt unregistered [ 4929.548862] Key type ._llcrypt registered [ 4929.552386] Key type .llcrypt registered [ 4929.737217] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 4931.435864] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4931.491952] alg: No test for adler32 (adler32-zlib) [ 4933.024298] Lustre: Lustre: Build Version: 2.17.51_1_gce01c51 [ 4933.302556] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 4935.036096] Key type lgssc registered [ 4937.001623] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4948.265513] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4948.291494] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4949.781531] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4956.218320] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4959.234440] LustreError: 114872:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4959.258381] LustreError: 114872:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 4964.322273] LustreError: 114873:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4967.720280] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4968.230711] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4973.804852] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4981.362375] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4985.665935] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4994.289181] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4994.908344] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5002.958334] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5010.191779] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5019.465642] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 5028.781278] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 5034.547916] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 5034.555344] Lustre: Skipped 2 previous similar messages [ 5036.715550] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 5036.797204] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 5041.113447] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 5041.135745] Lustre: Skipped 2 previous similar messages [ 5042.525884] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 5042.553995] Lustre: Skipped 1 previous similar message [ 5054.827808] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.201.5@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 5054.835181] Lustre: Skipped 2 previous similar messages [ 5060.668197] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.201.5@tcp 192.168.0.[2,10]@tcp [ 5068.403653] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.201.5/24@tcp in log lustre-MDT0000 [ 5068.440721] Lustre: Skipped 2 previous similar messages [ 5070.758849] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.5/24@tcp [ 5070.785297] Lustre: Skipped 1 previous similar message [ 5085.157349] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5085.174849] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5085.199919] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5086.186329] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5086.190597] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5086.220387] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5087.685623] Lustre: server umount lustre-OST0000 complete [ 5096.421066] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5096.421892] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5096.461682] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5096.468050] LustreError: Skipped 5 previous similar messages [ 5096.491593] Lustre: Skipped 1 previous similar message [ 5098.614941] Lustre: server umount lustre-MDT0000 complete [ 5101.540024] LustreError: 114873:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5101.582538] LustreError: 114873:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 5103.374491] LustreError: 117807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773612243 with bad export cookie 10587764491448944921 [ 5103.378419] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5103.395494] LustreError: 117807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5103.815964] Lustre: server umount lustre-MDT0001 complete [ 5117.799190] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 5120.918590] Key type lgssc unregistered [ 5121.400735] LNet: 119300:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5121.414880] LNetError: 119300:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5121.440133] LNet: Removed LNI 192.168.201.105@tcp [ 5122.687428] Key type .llcrypt unregistered [ 5122.689973] Key type ._llcrypt unregistered [ 5159.192853] Key type ._llcrypt registered [ 5159.198132] Key type .llcrypt registered [ 5159.369435] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 5160.843050] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5160.896855] alg: No test for adler32 (adler32-zlib) [ 5162.137434] Lustre: Lustre: Build Version: 2.17.51_1_gce01c51 [ 5162.502267] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 5164.248614] Key type lgssc registered [ 5165.483858] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5174.373228] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5174.402700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5176.240418] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 5176.277171] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.201.5/24@tcp [ 5176.281532] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5182.404673] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5186.536047] LustreError: 120848:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5186.545545] LustreError: 120848:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 5191.654581] LustreError: 120847:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5193.746523] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5194.173962] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 5194.204804] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.5/24@tcp [ 5194.273311] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5199.487953] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5206.695451] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5212.332587] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5220.686761] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5221.213083] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5222.437104] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:169 to 0x240000401:193) [ 5229.418900] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5237.218919] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5253.091870] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5253.124817] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5253.175950] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5255.138268] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5255.139238] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5255.146133] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5255.184779] Lustre: Skipped 1 previous similar message [ 5256.845101] Lustre: server umount lustre-OST0000 complete [ 5265.380611] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5265.382234] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5265.411055] Lustre: Skipped 1 previous similar message [ 5265.434520] Lustre: Skipped 1 previous similar message [ 5268.574555] Lustre: server umount lustre-MDT0000 complete [ 5270.521026] LustreError: 120848:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5270.551522] LustreError: 120848:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 5274.478121] LustreError: 120828:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773612414 with bad export cookie 16797526361950413590 [ 5274.488916] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5274.501352] LustreError: 120828:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5274.962961] Lustre: server umount lustre-MDT0001 complete [ 5289.330856] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 5292.898910] Key type lgssc unregistered [ 5293.320956] LNet: 124085:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5293.325418] LNetError: 124085:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5294.374478] LNet: Removed LNI 192.168.201.105@tcp [ 5295.421276] Key type .llcrypt unregistered [ 5295.424195] Key type ._llcrypt unregistered [ 5330.951252] Key type ._llcrypt registered [ 5330.954583] Key type .llcrypt registered [ 5331.072682] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 5349.497449] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 5351.063687] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5351.253837] alg: No test for adler32 (adler32-zlib) [ 5352.402928] Lustre: Lustre: Build Version: 2.17.51_1_gce01c51 [ 5352.699879] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 5354.368175] Key type lgssc registered [ 5355.905769] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5398.848974] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5398.886269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5400.153912] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5400.179842] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5400.309169] Lustre: lustre-MDT0000: new disk, initializing [ 5400.459124] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5400.481508] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5405.529101] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5419.798130] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5420.014726] Lustre: 127455:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5420.092893] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5420.096458] Lustre: Skipped 1 previous similar message [ 5420.205807] Lustre: lustre-MDT0001: new disk, initializing [ 5420.310322] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5420.368114] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5420.377339] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5425.436860] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5434.540429] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5440.327204] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5449.328040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5449.710881] Lustre: lustre-OST0000: new disk, initializing [ 5449.715662] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5449.869319] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5456.508823] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5459.024792] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5459.034972] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5459.154548] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5464.968374] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5470.817694] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5471.104375] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5477.229434] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5477.504631] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5479.909416] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5479.912872] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5479.916395] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5479.938121] Lustre: Skipped 1 previous similar message [ 5485.026327] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5485.039504] Lustre: Skipped 2 previous similar messages [ 5490.154087] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5490.161062] Lustre: Skipped 1 previous similar message [ 5494.240604] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5494.396351] Lustre: server umount lustre-OST0000 complete [ 5500.386101] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5500.401265] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5500.420663] Lustre: Skipped 1 previous similar message [ 5500.439435] Lustre: Skipped 1 previous similar message [ 5505.357878] Lustre: server umount lustre-MDT0000 complete [ 5510.043726] LustreError: 127446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773612650 with bad export cookie 936918477591712960 [ 5510.074572] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5510.444459] Lustre: server umount lustre-MDT0001 complete [ 5520.521623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5521.040937] LustreError: 131273:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5521.067938] LustreError: 131273:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 5521.148652] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5526.174353] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5526.501823] LustreError: 131274:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5531.626260] LustreError: 131273:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5535.264523] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5535.522311] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5540.614967] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5547.452587] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5552.355386] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5560.539350] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5561.056877] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5569.988439] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5577.796561] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5583.772119] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5584.210771] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5589.290227] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5589.538815] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5601.726859] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5602.003776] Lustre: lustre-OST0001: new disk, initializing [ 5602.009468] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5602.177685] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5610.563603] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5611.607057] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5611.617583] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5611.742698] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5619.826382] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5620.192686] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5626.386716] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5626.783801] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5632.118939] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8b966064c800.ost_server_uuid 50 [ 5634.525039] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8b966064c800.ost_server_uuid in IDLE state after 0 sec [ 5647.853452] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5647.855222] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5647.882529] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5647.886777] LustreError: Skipped 1 previous similar message [ 5647.899723] Lustre: Skipped 1 previous similar message [ 5650.432650] Lustre: server umount lustre-OST0001 complete [ 5652.985682] LustreError: 132870:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5653.000904] LustreError: 132870:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 5654.502548] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5654.515733] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5654.537273] Lustre: Skipped 1 previous similar message [ 5658.088177] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5658.102872] Lustre: Skipped 1 previous similar message [ 5658.113430] LustreError: 132870:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5658.155353] LustreError: 132870:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 5660.687620] Lustre: server umount lustre-OST0000 complete [ 5668.323612] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5668.343722] Lustre: Skipped 1 previous similar message [ 5670.965477] Lustre: server umount lustre-MDT0000 complete [ 5673.447312] LustreError: 131274:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5675.483253] LustreError: 131255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773612815 with bad export cookie 936918477591713842 [ 5675.487077] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5675.491023] LustreError: 131255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5675.952703] Lustre: server umount lustre-MDT0001 complete [ 5699.549850] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 5702.362961] Key type lgssc unregistered [ 5702.728592] LNet: 137118:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5702.741958] LNetError: 137118:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5702.768796] LNet: Removed LNI 192.168.201.105@tcp [ 5703.623748] Key type .llcrypt unregistered [ 5703.627897] Key type ._llcrypt unregistered [ 5747.129194] Key type ._llcrypt registered [ 5747.131890] Key type .llcrypt registered [ 5747.247359] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 5763.023383] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 5764.528953] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5764.543639] alg: No test for adler32 (adler32-zlib) [ 5765.596740] Lustre: Lustre: Build Version: 2.17.51_1_gce01c51 [ 5765.837665] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 5767.528250] Key type lgssc registered [ 5768.667422] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5804.510850] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5804.527350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5805.881766] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5805.912203] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5806.006295] Lustre: lustre-MDT0000: new disk, initializing [ 5806.123611] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5806.152244] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5809.873949] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5821.151582] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5821.298856] Lustre: 141239:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5821.356321] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5821.359977] Lustre: Skipped 1 previous similar message [ 5821.428925] Lustre: lustre-MDT0001: new disk, initializing [ 5821.535805] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5821.593268] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5821.608539] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5826.615910] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5834.178752] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5837.829817] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5844.012753] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5844.271815] Lustre: lustre-OST0000: new disk, initializing [ 5844.280071] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5844.392740] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5849.283877] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5852.779717] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5852.802562] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5852.910071] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5857.366823] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5861.900737] Lustre: Failing over lustre-OST0000 [ 5862.137887] Lustre: server umount lustre-OST0000 complete [ 5879.777029] Lustre: 139467:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773613003/real 1773613003] req@00000000f426e218 x1859767925505024/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1773613019 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5879.777322] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5879.811520] Lustre: 139467:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 5883.039952] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5883.450649] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5883.478709] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 5884.465098] Lustre: 139468:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773613008/real 1773613008] req@000000006ac0e2c0 x1859767925506048/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1773613024 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5884.509635] Lustre: 139468:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 5888.999397] Lustre: 139465:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773613013/real 1773613013] req@00000000e7e44974 x1859767925507328/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1773613029 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5889.030027] Lustre: 139465:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 5889.258879] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5889.573052] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 5890.734401] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 5890.737354] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5893.409464] Lustre: Failing over lustre-MDT0000 [ 5893.815338] Lustre: server umount lustre-MDT0000 complete [ 5894.432195] Lustre: 139465:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773613018/real 1773613018] req@00000000bce51650 x1859767925508352/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1773613034 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5894.450330] Lustre: 139465:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 5894.627833] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5894.656809] Lustre: Skipped 3 previous similar messages [ 5899.746947] LustreError: 142429:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5899.767058] LustreError: 142429:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 7 previous similar messages [ 5900.926148] LustreError: 141247:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5904.867588] LustreError: 141252:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5904.895484] LustreError: 141252:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 5909.984110] Lustre: 139466:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773613034/real 1773613034] req@00000000cf826d98 x1859767925518336/t0(0) o400->MGC192.168.201.105@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1773613050 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5909.993208] LustreError: 141251:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5910.032979] Lustre: 139466:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 5910.034045] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5910.086610] LustreError: 141251:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 5 previous similar messages [ 5913.422783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5914.085389] LustreError: 141252:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5914.113079] LustreError: 141252:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 4 previous similar messages [ 5920.229822] LustreError: 139464:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000f620d95a x1859767925524992/t0(0) o250->MGC192.168.201.105@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5920.616887] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5920.678196] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5922.431525] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5925.532263] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5925.901641] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5925.910287] Lustre: Skipped 1 previous similar message [ 5925.939351] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 5926.000965] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5936.119511] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5936.123699] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5936.136174] Lustre: Skipped 1 previous similar message [ 5939.711989] Lustre: server umount lustre-OST0000 complete [ 5946.340071] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5946.340283] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5946.345863] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5946.345872] Lustre: Skipped 1 previous similar message [ 5946.385523] Lustre: Skipped 1 previous similar message [ 5949.944379] Lustre: server umount lustre-MDT0000 complete [ 5951.457307] LustreError: 141252:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5951.478886] LustreError: 141252:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 9 previous similar messages [ 5953.850947] LustreError: 144124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773613093 with bad export cookie 1308139572504431686 [ 5953.855436] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5953.863588] LustreError: 144124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5954.249655] Lustre: server umount lustre-MDT0001 complete [ 5966.270256] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 5969.333358] Key type lgssc unregistered [ 5969.648281] LNet: 146485:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5969.656519] LNetError: 146485:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5969.678664] LNet: Removed LNI 192.168.201.105@tcp [ 5970.524729] Key type .llcrypt unregistered [ 5970.531330] Key type ._llcrypt unregistered [ 6010.002260] Key type ._llcrypt registered [ 6010.005301] Key type .llcrypt registered [ 6010.145886] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 6011.868946] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6011.975458] alg: No test for adler32 (adler32-zlib) [ 6013.054516] Lustre: Lustre: Build Version: 2.17.51_1_gce01c51 [ 6013.336905] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 6015.072858] Key type lgssc registered [ 6016.461378] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6024.387483] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6024.405674] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6025.886763] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6030.558259] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6035.432779] LustreError: 147991:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6035.447832] LustreError: 147991:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 6039.183303] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6040.551167] LustreError: 147992:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6043.599389] LustreError: 148020:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 6043.612364] LustreError: 148020:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 6043.621689] LustreError: 148020:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 0, failed: rc = -5 [ 6043.992924] Lustre: server umount lustre-MDT0000 complete [ 6059.178389] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 6071.861748] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 6114.152347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6114.448368] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6114.476128] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6114.565296] Lustre: lustre-MDT0000: new disk, initializing [ 6114.678055] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6114.700375] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6119.386768] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6130.177770] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6130.321613] Lustre: 151179:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6130.395980] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6130.400502] Lustre: Skipped 1 previous similar message [ 6130.482608] Lustre: lustre-MDT0001: new disk, initializing [ 6130.591385] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6130.629214] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6130.651387] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6134.318719] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6141.447060] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6145.315789] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6152.296259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6152.599868] Lustre: lustre-OST0000: new disk, initializing [ 6152.611430] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6152.710378] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6158.463428] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6159.436064] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6159.443714] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6159.543840] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6166.037642] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6170.294200] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 6172.298315] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 18:21:50 (1773613310) [ 6559.248871] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6559.251521] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6559.287677] Lustre: Skipped 2 previous similar messages [ 6559.296923] Lustre: Skipped 2 previous similar messages [ 6564.321763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6564.342033] Lustre: Skipped 2 previous similar messages [ 6565.892826] Lustre: server umount lustre-MDT0000 complete [ 6569.442883] LustreError: 154578:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6569.479161] LustreError: 154578:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 6570.770305] LustreError: 151172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773613710 with bad export cookie 5718696926563758541 [ 6570.774948] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6570.781981] LustreError: 151172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6574.560820] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6574.562699] LustreError: 151187:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6574.598861] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6580.711296] LustreError: 151187:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6580.712933] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6580.743760] LustreError: 151187:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 6580.777408] Lustre: Skipped 1 previous similar message [ 6585.313323] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6585.896984] Lustre: server umount lustre-MDT0001 complete [ 6598.913809] Lustre: server umount lustre-OST0000 complete [ 6615.691693] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 6634.382408] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 6693.386068] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6693.791124] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6693.866225] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6694.021812] Lustre: lustre-MDT0000: new disk, initializing [ 6694.168425] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6694.188721] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6701.762456] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6718.255455] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6718.536688] Lustre: 157490:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6718.651589] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6718.663206] Lustre: Skipped 1 previous similar message [ 6718.849780] Lustre: lustre-MDT0001: new disk, initializing [ 6719.010092] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6719.072648] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6719.086843] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6726.417625] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6737.605982] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6743.387602] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6754.628121] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6755.133651] Lustre: lustre-OST0000: new disk, initializing [ 6755.148988] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6755.317475] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6762.628363] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6762.639511] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6762.862984] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6766.191559] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6779.481882] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6787.937506] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6788.468523] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6797.894797] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6798.123502] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6803.960454] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6803.962309] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6803.984202] LustreError: Skipped 1 previous similar message [ 6804.004441] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6804.041151] Lustre: Skipped 1 previous similar message [ 6804.105574] Lustre: Skipped 1 previous similar message [ 6814.183617] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6814.196185] Lustre: Skipped 3 previous similar messages [ 6816.225922] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6816.610283] Lustre: server umount lustre-OST0000 complete [ 6824.418826] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6824.433404] Lustre: Skipped 1 previous similar message [ 6829.556269] LustreError: 157500:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6829.586634] LustreError: 157500:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 6829.919604] Lustre: server umount lustre-MDT0000 complete [ 6836.470243] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6836.471249] LustreError: 159244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773613976 with bad export cookie 5718696926563917378 [ 6836.509060] LustreError: 159244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6837.185478] Lustre: server umount lustre-MDT0001 complete [ 6853.585178] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 18:33:09 (1773613989) [ 6880.986132] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 6900.262563] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 6967.142883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6967.551695] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6967.588204] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6967.777753] Lustre: lustre-MDT0000: new disk, initializing [ 6967.991701] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6968.027436] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6975.001093] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6994.032858] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6994.297278] Lustre: 163848:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6995.065856] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7002.867625] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7016.064797] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7023.116888] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7034.280198] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7035.005862] Lustre: lustre-OST0000: new disk, initializing [ 7035.014677] Lustre: Skipped 1 previous similar message [ 7035.027043] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7035.033332] Lustre: Skipped 2 previous similar messages [ 7035.270734] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7035.286065] Lustre: Skipped 1 previous similar message [ 7042.213900] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7042.230600] Lustre: Skipped 1 previous similar message [ 7042.239975] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7042.442983] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7045.065664] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7058.050432] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7071.212608] Lustre: server umount lustre-OST0000 complete [ 7078.371525] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7078.376330] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7078.403051] Lustre: Skipped 1 previous similar message [ 7078.424909] Lustre: Skipped 3 previous similar messages [ 7087.886720] Lustre: server umount lustre-MDT0000 complete [ 7088.614168] LustreError: 163854:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7088.636210] LustreError: 163854:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 7089.569912] Lustre: 147302:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773614213/real 1773614213] req@00000000e1bef8e3 x1859768186347904/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1773614229 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7089.609458] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7094.764135] Lustre: 147304:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773614218/real 1773614218] req@00000000e63232c5 x1859768186350080/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1773614234 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7094.817654] Lustre: 147304:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 7094.888943] LustreError: 163840:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773614234 with bad export cookie 5718696926563918260 [ 7094.897311] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7094.908394] LustreError: 163840:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7095.665259] Lustre: server umount lustre-MDT0001 complete [ 7111.111005] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 18:37:27 (1773614247) [ 7137.551227] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 7154.316549] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 7218.027072] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7218.614524] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7218.695087] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7218.930213] Lustre: lustre-MDT0000: new disk, initializing [ 7219.159872] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7219.203391] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7226.086333] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7243.519803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7243.830399] Lustre: 169807:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7244.426383] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7251.773274] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7264.576772] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7272.357750] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7284.481242] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7293.013701] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7293.260601] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7294.954483] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7306.857304] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7324.129961] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7324.163954] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7324.176055] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7324.176065] Lustre: Skipped 2 previous similar messages [ 7324.226910] LustreError: Skipped 1 previous similar message [ 7333.344545] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7333.652236] Lustre: server umount lustre-OST0000 complete [ 7345.896078] Lustre: server umount lustre-MDT0000 complete [ 7349.730263] LustreError: 169814:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7349.791657] LustreError: 169814:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 5 previous similar messages [ 7353.813751] LustreError: 169798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773614493 with bad export cookie 5718696926563919338 [ 7353.848037] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7354.622559] Lustre: server umount lustre-MDT0001 complete [ 7375.414330] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 7380.889663] Key type lgssc unregistered [ 7381.347809] LNet: 173313:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7381.364689] LNetError: 173313:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7381.398232] LNet: Removed LNI 192.168.201.105@tcp [ 7382.690539] Key type .llcrypt unregistered [ 7382.693675] Key type ._llcrypt unregistered [ 7430.878654] Key type ._llcrypt registered [ 7430.883769] Key type .llcrypt registered [ 7431.090442] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 7433.570940] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7433.635201] alg: No test for adler32 (adler32-zlib) [ 7435.019762] Lustre: Lustre: Build Version: 2.17.51_1_gce01c51 [ 7435.438767] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 7437.288365] Key type lgssc registered [ 7439.463233] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7452.216380] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7452.243615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7454.385566] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7461.239589] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7464.931591] LustreError: 174722:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7464.962868] LustreError: 174722:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 7470.054206] LustreError: 174721:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7474.604994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7474.932494] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7482.087764] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7490.736271] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7497.915500] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7509.718561] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7510.323523] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7520.394472] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7524.742627] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 7529.767650] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7555.541537] Lustre: server umount lustre-OST0000 complete [ 7565.796597] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7565.806415] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7565.838025] Lustre: Skipped 1 previous similar message [ 7565.852553] Lustre: Skipped 1 previous similar message [ 7570.918919] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7570.932120] Lustre: Skipped 1 previous similar message [ 7570.961911] Lustre: 174031:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773614695/real 1773614695] req@000000004e29103f x1859769676628992/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1773614711 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7570.993494] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7572.251587] Lustre: server umount lustre-MDT0000 complete [ 7576.036341] LustreError: 176882:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7576.041468] Lustre: 174030:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773614700/real 1773614700] req@00000000180771f5 x1859769676630912/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1773614716 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7576.079464] LustreError: 176882:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 7578.088652] LustreError: 177193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773614718 with bad export cookie 9502472948203927256 [ 7578.092541] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7578.111753] LustreError: 177193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7578.759940] Lustre: server umount lustre-MDT0001 complete [ 7599.840926] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 7603.982130] Key type lgssc unregistered [ 7604.416468] LNet: 177955:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7604.427334] LNetError: 177955:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7605.478698] LNet: Removed LNI 192.168.201.105@tcp [ 7606.772338] Key type .llcrypt unregistered [ 7606.778525] Key type ._llcrypt unregistered [ 7651.635840] Key type ._llcrypt registered [ 7651.638951] Key type .llcrypt registered [ 7651.919683] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 7655.133348] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7655.318793] alg: No test for adler32 (adler32-zlib) [ 7656.694287] Lustre: Lustre: Build Version: 2.17.51_1_gce01c51 [ 7657.059402] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 7658.864876] Key type lgssc registered [ 7661.143921] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7672.849686] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7672.887350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7675.140351] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7681.667906] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7685.603123] LustreError: 179359:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7685.668955] LustreError: 179359:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 7690.731565] LustreError: 179360:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7695.847634] LustreError: 179359:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7696.976040] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7697.803029] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7704.812115] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7714.363848] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7720.311303] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7730.682270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7731.265170] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7740.374624] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7745.674159] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 7749.988841] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7758.305963] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7758.322902] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7758.341032] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7761.381885] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7761.386932] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7761.419588] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7762.590497] Lustre: lustre-OST0000: Not available for connect from 192.168.201.5@tcp (stopping) [ 7762.606527] Lustre: Skipped 1 previous similar message [ 7764.499639] Lustre: server umount lustre-OST0000 complete [ 7774.116989] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8b96468cc800.ost_server_uuid 50 [ 7777.037383] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 7796.199202] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7796.235539] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7796.254646] Lustre: Skipped 1 previous similar message [ 7798.567786] Lustre: server umount lustre-MDT0000 complete [ 7801.316355] LustreError: 180072:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7801.333831] LustreError: 180072:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 7804.402478] LustreError: 179342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773614944 with bad export cookie 9397861469509857349 [ 7804.428602] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7804.429820] LustreError: 179342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7804.945100] Lustre: server umount lustre-MDT0001 complete [ 7820.987879] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 18:49:17 (1773614957) [ 7834.581536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7835.059233] LustreError: 182849:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7835.178972] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7842.852832] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7845.350899] LustreError: 182850:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7845.400215] LustreError: 182850:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 7856.129425] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7856.820371] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7864.387243] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7873.709730] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7880.687914] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7891.450354] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7892.149984] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7903.327911] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7905.481137] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7913.836538] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7925.687512] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7926.176692] Lustre: lustre-OST0001: new disk, initializing [ 7926.182976] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7926.388635] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7934.611422] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7934.623970] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7934.851090] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7935.989345] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7949.705878] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7955.443071] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7955.462318] Lustre: Skipped 1 previous similar message [ 7955.469797] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7955.479220] Lustre: Skipped 1 previous similar message [ 7960.806138] Lustre: server umount lustre-OST0000 complete [ 7961.216734] LustreError: 184959:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7961.258660] LustreError: 184959:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 7971.510929] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7971.978564] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7980.412717] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7980.676672] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7996.405680] LustreError: 184446:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7996.438982] LustreError: 184446:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 19 previous similar messages [ 8001.509652] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8001.526431] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8001.530732] LustreError: Skipped 1 previous similar message [ 8001.532240] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8001.532247] Lustre: Skipped 3 previous similar messages [ 8001.566181] Lustre: Skipped 2 previous similar messages [ 8002.112973] Lustre: server umount lustre-OST0001 complete [ 8010.218110] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8010.232092] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8015.757196] Lustre: server umount lustre-MDT0000 complete [ 8022.648510] LustreError: 182829:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773615162 with bad export cookie 9397861469509858420 [ 8022.668034] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8022.681127] LustreError: 182829:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8023.611444] Lustre: server umount lustre-MDT0001 complete [ 8067.451798] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 18:53:24 (1773615204) [ 8080.248457] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8080.721301] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8080.754802] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8080.992062] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8088.197993] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8102.087978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8102.524314] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8102.697583] Lustre: 188661:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8110.134526] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8118.512484] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8126.958534] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8139.818692] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8140.351710] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8149.754247] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8151.127446] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8161.051553] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8172.739624] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8172.904643] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 8173.459634] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8173.477803] Lustre: Skipped 2 previous similar messages [ 8185.384988] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8196.582430] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 8202.940896] Lustre: lustre-OST0000: Not available for connect from 192.168.201.5@tcp (stopping) [ 8202.958898] Lustre: Skipped 4 previous similar messages [ 8204.326862] Lustre: server umount lustre-OST0000 complete [ 8205.801940] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8205.831376] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8205.868337] Lustre: Skipped 1 previous similar message [ 8205.885331] LustreError: 190288:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8205.921345] LustreError: 190288:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 6 previous similar messages [ 8226.785289] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8226.814890] LustreError: Skipped 2 previous similar messages [ 8227.496371] Lustre: server umount lustre-OST0001 complete [ 8236.519769] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8236.535913] Lustre: Skipped 2 previous similar messages [ 8240.402473] Lustre: server umount lustre-MDT0000 complete [ 8248.417218] LustreError: 188651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773615388 with bad export cookie 9397861469509859827 [ 8248.428791] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8248.453166] LustreError: 188651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8249.282972] Lustre: server umount lustre-MDT0001 complete [ 8297.352461] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 18:57:14 (1773615434) [ 8325.133060] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 8341.097501] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 8401.667557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8402.008285] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8402.056518] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8402.177462] Lustre: lustre-MDT0000: new disk, initializing [ 8402.385941] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8402.423755] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8408.697597] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8425.899927] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8426.092474] Lustre: 196312:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8426.167188] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8426.172417] Lustre: Skipped 1 previous similar message [ 8426.253891] Lustre: lustre-MDT0001: new disk, initializing [ 8426.411303] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8426.428231] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8433.589569] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8447.137273] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8454.253696] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8465.414885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8465.817771] Lustre: lustre-OST0000: new disk, initializing [ 8465.823225] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8473.149574] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8473.171278] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8473.314387] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8475.736491] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8489.394827] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8498.519302] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8498.935661] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8506.622974] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8506.925043] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8514.531114] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8514.538214] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8514.551165] LustreError: Skipped 2 previous similar messages [ 8514.551848] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8514.551853] Lustre: Skipped 2 previous similar messages [ 8514.614346] Lustre: Skipped 6 previous similar messages [ 8516.131965] Lustre: server umount lustre-OST0000 complete [ 8529.681080] Lustre: server umount lustre-MDT0000 complete [ 8529.898802] LustreError: 196322:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8529.935200] LustreError: 196322:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 18 previous similar messages [ 8535.745449] LustreError: 197090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773615675 with bad export cookie 9397861469509861388 [ 8535.762822] LustreError: 197090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8535.765792] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8536.615597] Lustre: server umount lustre-MDT0001 complete [ 8551.616395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8560.211797] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8574.594886] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8583.199711] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8593.397275] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8600.452806] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8613.088744] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8624.741929] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8634.644001] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8642.890730] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8643.273596] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8650.269231] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8650.645052] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8653.800217] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8653.809739] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8653.837996] Lustre: Skipped 2 previous similar messages [ 8653.843480] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8653.843490] Lustre: Skipped 4 previous similar messages [ 8653.881378] LustreError: Skipped 1 previous similar message [ 8659.977230] Lustre: server umount lustre-OST0000 complete [ 8670.294369] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8677.862586] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8713.967509] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8714.513800] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8714.526939] Lustre: Skipped 5 previous similar messages [ 8725.338748] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8734.126335] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8734.142091] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8734.142226] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8734.171649] Lustre: Skipped 1 previous similar message [ 8736.791951] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8739.320040] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8749.543682] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8749.569580] LustreError: Skipped 1 previous similar message [ 8751.716692] Lustre: server umount lustre-OST0000 complete [ 8765.148635] Lustre: server umount lustre-MDT0000 complete [ 8773.554627] LustreError: 200126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773615913 with bad export cookie 9397861469509862263 [ 8773.578429] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8773.586210] LustreError: 200126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8775.251554] Lustre: server umount lustre-MDT0001 complete [ 8788.506986] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 19:05:25 (1773615925) [ 8798.305426] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8799.539227] LustreError: 205607:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8799.557031] LustreError: 205607:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 11 previous similar messages [ 8807.430701] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8823.592227] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8831.005603] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8840.418845] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8847.273796] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8859.256672] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8869.749857] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8879.000456] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8888.757820] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8889.499869] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8896.990948] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8897.447817] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8914.073614] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8914.496517] Lustre: lustre-OST0001: new disk, initializing [ 8914.507767] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8920.897164] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8920.923318] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8921.080522] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8929.899121] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8941.699243] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 8948.785402] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 8949.276625] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8956.520930] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 8957.033947] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8962.017786] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8962.018836] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8962.047256] LustreError: Skipped 2 previous similar messages [ 8962.049763] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8962.067485] Lustre: Skipped 6 previous similar messages [ 8962.101377] Lustre: Skipped 8 previous similar messages [ 8966.280907] Lustre: server umount lustre-OST0001 complete [ 8978.292883] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 8978.723251] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 8987.039665] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 8987.384933] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 9021.802600] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9027.592253] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 9027.644708] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 9027.652097] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 9027.720331] Lustre: Skipped 1 previous similar message [ 9034.318318] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 9044.580771] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 9047.780961] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [ 9057.037519] Lustre: server umount lustre-OST0001 complete [ 9084.185962] Lustre: server umount lustre-OST0000 complete [ 9096.530669] Lustre: server umount lustre-MDT0000 complete [ 9102.417044] LustreError: 205590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773616242 with bad export cookie 9397861469509863236 [ 9102.424825] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9102.454165] LustreError: 205590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9109.013854] Lustre: server umount lustre-MDT0001 complete [ 9146.182801] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 19:11:23 (1773616283) [ 9157.482349] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9157.878838] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9157.909942] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9165.155892] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 9178.423424] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9178.583584] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9178.618593] Lustre: 213582:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9185.615086] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 9194.746921] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9202.380986] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9212.172288] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9212.425904] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9223.479122] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 9233.661263] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9249.640946] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9249.929585] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9250.360645] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9250.375316] Lustre: Skipped 8 previous similar messages [ 9261.395789] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 9272.021867] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 9275.557375] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [ 9282.862680] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 9283.475473] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9292.185500] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 9292.700322] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9299.737886] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8b9648878800.ost_server_uuid 50 [ 9303.134583] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8b9648878800.ost_server_uuid in IDLE state after 0 sec [ 9305.237642] Lustre: Permanently deactivating lustre-OST0001 [ 9305.271069] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 9312.297543] Lustre: Permanently reactivating lustre-OST0001 [ 9312.347408] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 9312.368797] Lustre: Skipped 2 previous similar messages [ 9324.012283] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9324.042749] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9324.048154] Lustre: Skipped 7 previous similar messages [ 9324.076427] LustreError: Skipped 2 previous similar messages [ 9327.872265] Lustre: server umount lustre-OST0001 complete [ 9329.151544] LustreError: 216319:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9329.212056] LustreError: 216319:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 50 previous similar messages [ 9340.667733] Lustre: server umount lustre-OST0000 complete [ 9361.376155] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9361.741832] Lustre: server umount lustre-MDT0000 complete [ 9367.540901] LustreError: 213573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773616507 with bad export cookie 9397861469509864524 [ 9367.541956] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9367.550154] LustreError: 213573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9368.167529] Lustre: server umount lustre-MDT0001 complete [ 9410.152548] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 19:15:46 (1773616546) [ 9422.046242] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9422.669310] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9422.721678] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9422.734482] Lustre: Skipped 2 previous similar messages [ 9430.841245] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 9444.622219] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9444.746128] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9444.769721] Lustre: 219711:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9451.336411] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 9460.810991] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9467.631505] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9478.880937] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9479.329852] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9479.381613] Lustre: Permanently deactivating lustre-OST0000 [ 9479.398938] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 9488.540073] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 9489.020507] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 9497.818625] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9509.081377] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9509.148330] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9515.839968] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 9518.317611] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 9526.770553] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 9529.621493] Lustre: Permanently reactivating lustre-OST0000 [ 9529.661855] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 9529.677575] Lustre: Skipped 2 previous similar messages [ 9538.691544] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 49 seconds. I think it's dead, and I am evicting it. exp 000000008b3ec952, cur 1773616678 deadline 1773616659 last 1773616629 [ 9539.244164] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 9539.261498] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 9539.281596] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9539.412491] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 9546.723797] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9546.744578] Lustre: Skipped 23 previous similar messages [ 9548.785930] Lustre: server umount lustre-OST0001 complete [ 9560.165709] Lustre: server umount lustre-OST0000 complete [ 9572.168691] Lustre: server umount lustre-MDT0000 complete [ 9577.579543] LustreError: 219702:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773616717 with bad export cookie 9397861469509866204 [ 9577.594497] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9577.613667] LustreError: 219702:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9578.079238] Lustre: server umount lustre-MDT0001 complete [ 9592.835316] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 9596.251494] Key type lgssc unregistered [ 9596.754922] LNet: 224127:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9596.764723] LNetError: 224127:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9596.793527] LNet: Removed LNI 192.168.201.105@tcp [ 9597.899454] Key type .llcrypt unregistered [ 9597.905857] Key type ._llcrypt unregistered [ 9627.603983] Key type ._llcrypt registered [ 9627.607663] Key type .llcrypt registered [ 9627.747490] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 9629.093678] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9629.183729] alg: No test for adler32 (adler32-zlib) [ 9630.930661] Lustre: Lustre: Build Version: 2.17.51_1_gce01c51 [ 9631.278148] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 9633.026212] Key type lgssc registered [ 9634.522416] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9648.700458] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9648.732023] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9650.704103] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9656.430819] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 9660.913849] LustreError: 225582:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9660.934797] LustreError: 225582:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 9666.051180] LustreError: 225581:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9668.355265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9668.668882] Lustre: Found index 1 for lustre-MDT0001, updating log [ 9668.751845] Lustre: 225569:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9668.766309] Lustre: Permanently deactivating lustre-MDT0001 [ 9668.784228] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 9669.349189] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9675.049542] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 9675.819676] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 9683.064748] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9687.965820] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9698.196860] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9698.842831] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9708.361877] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 9714.150190] LustreError: 227196:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9717.275088] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9719.284263] LustreError: 227196:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9720.356245] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 9725.413698] LustreError: 227709:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9725.431162] LustreError: 227709:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 9727.311826] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9727.620618] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9733.115266] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 9733.132150] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 9735.183132] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 9743.208778] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 9746.733734] Lustre: Permanently reactivating lustre-MDT0001 [ 9746.813929] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 9746.819297] Lustre: Skipped 2 previous similar messages [ 9748.789259] Lustre: lustre-MDT0001: Client e3cd3c9b-8da1-4b1b-8b4b-d6586d72af68 (at 192.168.201.5@tcp) reconnecting [ 9753.737354] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9753.762299] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 9753.778222] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 9753.786883] LustreError: 228770:0:(llog_osd.c:2259:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 9753.793822] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 9753.804326] LustreError: 228770:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 9753.806391] LustreError: 228770:0:(obd_config.c:2136:class_config_llog_handler()) MGC192.168.201.105@tcp: cfg command failed: rc = -108 [ 9753.848370] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 9753.848370] [ 9753.870972] LustreError: 225573:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 9754.110835] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9756.480729] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9764.126568] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 9764.468306] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 9770.186649] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 9770.469741] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 9774.082213] Lustre: Permanently deactivating lustre-MDT0001 [ 9774.111657] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 9774.121803] Lustre: Skipped 2 previous similar messages [ 9782.533262] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 9794.562972] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9794.568711] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9794.586508] Lustre: Skipped 2 previous similar messages [ 9794.605673] Lustre: Skipped 3 previous similar messages [ 9794.989759] Lustre: server umount lustre-MDT0000 complete [ 9799.657727] LustreError: 225578:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9799.710044] LustreError: 225578:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 9801.331349] LustreError: 225561:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773616941 with bad export cookie 3383694676484382188 [ 9801.338707] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9801.359302] LustreError: 225561:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9802.287781] Lustre: server umount lustre-MDT0001 complete [ 9807.212995] Lustre: server umount lustre-OST0000 complete [ 9822.335446] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 19:22:39 (1773616959) [ 9837.687716] Lustre: server umount lustre-OST0001 complete [ 9848.050247] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 9861.878896] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 9912.494491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9912.855902] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9912.868500] Lustre: Skipped 2 previous similar messages [ 9912.906454] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9913.023823] Lustre: lustre-MDT0000: new disk, initializing [ 9913.144494] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9913.180812] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9919.094213] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 9932.818511] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9932.938847] Lustre: 233285:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9932.992445] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9932.997752] Lustre: Skipped 1 previous similar message [ 9933.081699] Lustre: lustre-MDT0001: new disk, initializing [ 9933.224875] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9933.292749] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9933.300898] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9938.098604] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 9947.127537] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9952.191424] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9961.006479] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9961.382611] Lustre: lustre-OST0000: new disk, initializing [ 9961.390112] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9961.517207] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9967.738056] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9967.775070] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9967.899861] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9968.955528] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 9978.801159] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9981.551207] LustreError: 234578:0:(mdt_handler.c:3251:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 9991.195608] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9991.553167] LustreError: 234578:0:(mdt_handler.c:3251:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 9991.581465] Lustre: lustre-OST0001: new disk, initializing [ 9991.600190] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9991.619916] LustreError: 235686:0:(mdt_handler.c:3251:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [10000.338671] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10001.030686] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10001.050541] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10001.217870] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10001.648132] LustreError: 235686:0:(mdt_handler.c:3251:mdt_reint_internal()) cfs_fail_timeout id 142 awake [10009.853713] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [10011.751262] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [10016.737551] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10016.765045] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10016.769333] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10016.779082] Lustre: Skipped 2 previous similar messages [10016.810157] Lustre: Skipped 1 previous similar message [10020.817169] Lustre: server umount lustre-OST0001 complete [10021.878698] LustreError: 235013:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10021.901324] LustreError: 235013:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [10042.365300] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10042.367278] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10042.394134] Lustre: Skipped 1 previous similar message [10042.411502] Lustre: Skipped 1 previous similar message [10047.464058] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10047.484063] Lustre: Skipped 1 previous similar message [10052.580360] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10052.605358] Lustre: Skipped 1 previous similar message [10054.112603] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10054.367371] Lustre: server umount lustre-OST0000 complete [10061.283607] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10061.298761] LustreError: Skipped 1 previous similar message [10061.308113] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10061.350155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10065.898703] Lustre: server umount lustre-MDT0000 complete [10067.950800] LustreError: 233295:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10067.989472] LustreError: 233295:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 13 previous similar messages [10071.471590] LustreError: 233278:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773617211 with bad export cookie 3383694676484384785 [10071.482876] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10071.488256] LustreError: 233278:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10078.068193] Lustre: server umount lustre-MDT0001 complete [10092.976886] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [10096.220558] Key type lgssc unregistered [10096.573967] LNet: 237938:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10096.583842] LNetError: 237938:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10096.617860] LNet: Removed LNI 192.168.201.105@tcp [10097.631667] Key type .llcrypt unregistered [10097.637183] Key type ._llcrypt unregistered [10146.198281] Key type ._llcrypt registered [10146.201253] Key type .llcrypt registered [10146.328253] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [10147.715405] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10147.919321] alg: No test for adler32 (adler32-zlib) [10148.976094] Lustre: Lustre: Build Version: 2.17.51_1_gce01c51 [10149.187356] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [10150.872187] Key type lgssc registered [10152.775245] Lustre: Echo OBD driver; http://www.lustre.org/ [10163.286265] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10163.303137] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10164.696857] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10164.761906] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10165.186233] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10171.137699] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10182.625948] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10182.832208] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10182.905697] Lustre: 240132:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10183.267294] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10189.770837] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10198.389563] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10203.628213] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10213.059066] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10213.318923] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10213.603487] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10220.333181] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [10222.215121] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10229.126872] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10292.202584] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10292.203765] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10292.205168] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10292.266735] Lustre: Skipped 1 previous similar message [10295.213972] Lustre: server umount lustre-OST0000 complete [10301.561945] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [10318.205971] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10318.623679] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10327.211488] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10331.964278] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [10331.978740] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [10331.999029] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [10332.037065] Lustre: Skipped 1 previous similar message [10334.825536] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10378.229514] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10378.248350] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10378.274286] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10378.282753] Lustre: Skipped 1 previous similar message [10383.336635] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10383.344519] Lustre: Skipped 1 previous similar message [10383.920368] Lustre: server umount lustre-OST0000 complete [10393.066333] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10393.087265] LustreError: Skipped 1 previous similar message [10393.098218] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10393.132582] Lustre: Skipped 1 previous similar message [10393.144405] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10398.692311] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10398.699557] Lustre: Skipped 2 previous similar messages [10403.297159] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10403.602068] Lustre: server umount lustre-MDT0000 complete [10408.874342] LustreError: 240126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773617548 with bad export cookie 1085566594301677913 [10408.880613] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10408.901684] LustreError: 240126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10409.505788] Lustre: server umount lustre-MDT0001 complete [10423.822464] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [10427.528763] Key type lgssc unregistered [10427.946397] LNet: 245097:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10427.962763] LNetError: 245097:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10427.979732] LNet: Removed LNI 192.168.201.105@tcp [10428.831346] Key type .llcrypt unregistered [10428.839759] Key type ._llcrypt unregistered [10455.728618] Key type ._llcrypt registered [10455.730975] Key type .llcrypt registered [10455.888083] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [10457.358706] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10457.430849] alg: No test for adler32 (adler32-zlib) [10458.612629] Lustre: Lustre: Build Version: 2.17.51_1_gce01c51 [10458.901732] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [10460.584224] Key type lgssc registered [10461.859688] Lustre: Echo OBD driver; http://www.lustre.org/ [10470.654782] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10470.679730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10472.400513] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10476.713385] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10482.667698] LustreError: 246499:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10482.690275] LustreError: 246499:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [10484.937107] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10485.298530] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10489.918156] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10496.182615] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10500.089466] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10506.409851] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10506.809192] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10512.422519] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10517.108197] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [10518.308889] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10546.658424] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10546.683801] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10546.707563] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10548.196877] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10548.199893] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10549.549922] Lustre: server umount lustre-OST0000 complete [10556.898938] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10556.907798] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10556.933283] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10556.946346] Lustre: Skipped 1 previous similar message [10559.752176] Lustre: server umount lustre-MDT0000 complete [10563.555663] LustreError: 246498:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10563.571888] LustreError: 246498:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [10563.608664] LustreError: 246477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773617703 with bad export cookie 11943787345407602318 [10563.626925] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10564.019859] Lustre: server umount lustre-MDT0001 complete [10574.213608] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [10576.952279] Key type lgssc unregistered [10577.305190] LNet: 250449:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10577.322358] LNetError: 250449:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10577.348813] LNet: Removed LNI 192.168.201.105@tcp [10578.259690] Key type .llcrypt unregistered [10578.265832] Key type ._llcrypt unregistered [10594.055578] Key type ._llcrypt registered [10594.058529] Key type .llcrypt registered [10594.201384] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [10595.030445] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10595.178063] alg: No test for adler32 (adler32-zlib) [10596.253449] Lustre: Lustre: Build Version: 2.17.51_1_gce01c51 [10596.501603] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [10598.184274] Key type lgssc registered [10599.518467] Lustre: Echo OBD driver; http://www.lustre.org/ [10606.981476] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10607.002720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10608.512650] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10612.649449] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10618.858880] LustreError: 251655:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10618.880554] LustreError: 251655:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [10619.855584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10620.154398] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10623.615876] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10627.644698] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10631.133066] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10637.238914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10637.516762] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10642.450890] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10647.180514] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10653.881536] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10666.469100] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10666.485711] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10666.501056] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10669.551271] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10669.562341] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10669.570830] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10672.157846] Lustre: server umount lustre-OST0000 complete [10676.705227] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10676.721404] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10676.739906] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10676.750657] Lustre: Skipped 1 previous similar message [10679.777670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10679.778582] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10681.390377] Lustre: server umount lustre-MDT0000 complete [10684.332711] LustreError: 251634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773617824 with bad export cookie 6020022322972444350 [10684.339294] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10684.347728] LustreError: 251634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10684.570919] Lustre: server umount lustre-MDT0001 complete [10692.689530] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [10695.631244] Key type lgssc unregistered [10695.955806] LNet: 255001:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10695.960641] LNetError: 255001:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10695.976388] LNet: Removed LNI 192.168.201.105@tcp [10696.768163] Key type .llcrypt unregistered [10696.773887] Key type ._llcrypt unregistered [10717.912960] Key type ._llcrypt registered [10717.918983] Key type .llcrypt registered [10718.050940] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [10719.116661] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10719.180179] alg: No test for adler32 (adler32-zlib) [10720.207016] Lustre: Lustre: Build Version: 2.17.51_1_gce01c51 [10720.368579] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [10722.024195] Key type lgssc registered [10723.131697] Lustre: Echo OBD driver; http://www.lustre.org/ [10729.553054] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10729.574572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10730.981921] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10734.627893] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10741.223846] LustreError: 256398:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10741.237469] LustreError: 256398:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [10742.028627] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10742.347438] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10746.427867] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10752.016625] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10755.404330] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10761.466591] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10761.771282] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10766.618488] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10771.360481] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10776.117831] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [10796.532808] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10796.540102] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10796.558606] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10796.583721] Lustre: Skipped 1 previous similar message [10801.419342] Lustre: server umount lustre-OST0000 complete [10806.757588] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10806.762601] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10806.768358] Lustre: Skipped 2 previous similar messages [10806.779332] Lustre: Skipped 1 previous similar message [10810.603988] Lustre: server umount lustre-MDT0000 complete [10811.874248] LustreError: 256398:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10813.855639] LustreError: 256380:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773617953 with bad export cookie 11252144413929241892 [10813.861144] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10813.867030] LustreError: 256380:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10814.131912] Lustre: server umount lustre-MDT0001 complete [10823.231654] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [10825.561965] Key type lgssc unregistered [10825.837503] LNet: 260351:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10825.842963] LNetError: 260351:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10825.857395] LNet: Removed LNI 192.168.201.105@tcp [10826.631374] Key type .llcrypt unregistered [10826.633574] Key type ._llcrypt unregistered [10839.883861] Key type ._llcrypt registered [10839.887208] Key type .llcrypt registered [10839.974360] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [10840.753943] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10840.767784] alg: No test for adler32 (adler32-zlib) [10841.792844] Lustre: Lustre: Build Version: 2.17.51_1_gce01c51 [10842.000485] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [10843.656193] Key type lgssc registered [10844.511312] Lustre: Echo OBD driver; http://www.lustre.org/ [10850.144802] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10850.167981] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10851.633572] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10854.891293] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10861.025505] LustreError: 261550:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10861.041587] LustreError: 261550:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [10861.141621] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10861.437419] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10864.434277] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10867.985344] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10870.105622] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10874.235196] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10874.461157] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10876.538614] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [10878.439718] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10882.442123] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10894.825276] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [10894.830667] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10894.845682] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10897.381168] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10897.387071] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10897.387378] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10897.423929] Lustre: Skipped 1 previous similar message [10899.312964] Lustre: server umount lustre-OST0000 complete [10907.620848] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10907.621675] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10907.636903] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10907.638809] Lustre: Skipped 1 previous similar message [10909.302812] Lustre: server umount lustre-MDT0000 complete [10912.237465] LustreError: 261516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773618052 with bad export cookie 8430472032360453817 [10912.241060] LustreError: 261516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10912.241476] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10912.417395] Lustre: server umount lustre-MDT0001 complete [10917.809377] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [10919.403684] Key type lgssc unregistered [10919.578763] LNet: 264895:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10919.583654] LNetError: 264895:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10919.593679] LNet: Removed LNI 192.168.201.105@tcp [10920.057185] Key type .llcrypt unregistered [10920.059058] Key type ._llcrypt unregistered [10926.485355] Key type ._llcrypt registered [10926.487710] Key type .llcrypt registered [10926.554693] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [10935.790329] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [10944.239784] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [10944.697947] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10944.731753] alg: No test for adler32 (adler32-zlib) [10945.663510] Lustre: Lustre: Build Version: 2.17.51_1_gce01c51 [10945.814027] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [10947.432186] Key type lgssc registered [10948.084548] Lustre: Echo OBD driver; http://www.lustre.org/ [10969.039190] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10969.064423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10970.255630] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10970.283567] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10970.342739] Lustre: lustre-MDT0000: new disk, initializing [10970.402673] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10970.423894] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10972.957787] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10979.818298] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10979.902729] Lustre: 268372:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10979.940764] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10979.943950] Lustre: Skipped 1 previous similar message [10979.997781] Lustre: lustre-MDT0001: new disk, initializing [10980.070466] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10980.096121] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10980.105833] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10982.449416] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10987.562452] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10989.719322] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10993.785136] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10993.948907] Lustre: lustre-OST0000: new disk, initializing [10993.952514] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10993.997523] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10994.331562] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10994.335799] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10994.407332] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10997.833093] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11002.781503] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11005.279795] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11005.423630] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11007.950821] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11008.099670] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11010.019653] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11010.022154] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11010.026872] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11010.035351] Lustre: Skipped 1 previous similar message [11010.043939] Lustre: Skipped 1 previous similar message [11015.141070] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11015.145672] Lustre: Skipped 1 previous similar message [11015.434841] Lustre: server umount lustre-OST0000 complete [11020.257976] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11020.258955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11020.268123] Lustre: Skipped 1 previous similar message [11024.068899] Lustre: server umount lustre-MDT0000 complete [11026.147551] LustreError: 268364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773618166 with bad export cookie 7136178782002241769 [11026.152601] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11026.154847] LustreError: 268364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11026.361988] Lustre: server umount lustre-MDT0001 complete [11031.364833] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 19:42:50 (1773618170) [11035.466479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11035.778930] LustreError: 272377:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11035.794890] LustreError: 272377:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [11035.833903] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11038.212893] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11041.250813] LustreError: 272378:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11042.930406] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11043.164329] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11045.314148] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11048.292024] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11050.607828] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11054.569373] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11054.855502] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11057.873287] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11061.133333] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11073.507535] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11073.507611] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11073.509306] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11073.509312] Lustre: Skipped 1 previous similar message [11073.514156] Lustre: Skipped 1 previous similar message [11073.517916] LustreError: Skipped 1 previous similar message [11077.035108] Lustre: server umount lustre-OST0000 complete [11079.139485] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11079.146034] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11079.156018] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11079.163193] Lustre: Skipped 1 previous similar message [11081.410030] Lustre: server umount lustre-MDT0000 complete [11083.621583] LustreError: 272358:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773618223 with bad export cookie 7136178782002242651 [11083.630368] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11083.630448] LustreError: 272358:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11083.833612] Lustre: server umount lustre-MDT0001 complete [11089.579852] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [11091.092378] Key type lgssc unregistered [11091.276501] LNet: 275556:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11091.281451] LNetError: 275556:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11091.291560] LNet: Removed LNI 192.168.201.105@tcp [11091.719606] Key type .llcrypt unregistered [11091.721137] Key type ._llcrypt unregistered [11107.062863] Key type ._llcrypt registered [11107.065473] Key type .llcrypt registered [11107.141889] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [11116.251767] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [11116.742425] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11116.804080] alg: No test for adler32 (adler32-zlib) [11117.728729] Lustre: Lustre: Build Version: 2.17.51_1_gce01c51 [11117.862300] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [11119.496353] Key type lgssc registered [11120.142987] Lustre: Echo OBD driver; http://www.lustre.org/ [11149.257737] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11149.275764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11150.480171] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11150.507210] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11150.567581] Lustre: lustre-MDT0000: new disk, initializing [11150.634734] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11150.652145] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11153.408802] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11161.256465] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11161.347466] Lustre: 279323:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11161.382297] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11161.385160] Lustre: Skipped 1 previous similar message [11161.443746] Lustre: lustre-MDT0001: new disk, initializing [11161.537554] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11161.578439] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11161.587493] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11164.537360] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11169.819785] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11172.316122] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11176.683707] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11176.872046] Lustre: lustre-OST2710: new disk, initializing [11176.875779] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [11176.934525] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [11178.028186] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [11178.034749] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [11178.139403] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [11180.813555] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11185.879818] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [11190.207545] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11190.368523] Lustre: lustre-OST03e8: new disk, initializing [11190.375704] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [11190.439206] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [11194.499049] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11198.184444] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [11199.009917] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [11199.016682] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [11199.082970] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [11203.017617] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [11203.190073] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11206.104215] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [11206.294231] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11209.207385] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [11209.345148] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11211.983602] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [11212.131505] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11217.889203] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11217.896685] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11217.907635] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11219.430352] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11219.431615] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11219.437446] Lustre: Skipped 1 previous similar message [11219.447604] Lustre: Skipped 3 previous similar messages [11220.561669] Lustre: server umount lustre-MDT0000 complete [11222.813691] LustreError: 279315:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773618362 with bad export cookie 862918855988457796 [11222.828103] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11223.039184] Lustre: server umount lustre-MDT0001 complete [11235.562783] Lustre: server umount lustre-OST2710 complete [11247.105665] Lustre: server umount lustre-OST03e8 complete [11259.172872] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [11266.052579] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [11291.361584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11291.509727] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11291.530113] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11291.581854] Lustre: lustre-MDT0000: new disk, initializing [11291.659727] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11291.675722] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11294.198392] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11300.933740] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11301.022904] Lustre: 287042:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11301.061416] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11301.068308] Lustre: Skipped 1 previous similar message [11301.128846] Lustre: lustre-MDT0001: new disk, initializing [11301.202595] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11301.234252] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11301.240510] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11303.573575] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11308.824730] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11311.256941] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11315.712865] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11316.141875] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11316.220097] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11319.311857] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11324.310053] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11326.864592] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11327.018564] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11329.806856] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11329.937932] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11331.561872] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11331.567589] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11331.572343] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11331.574196] Lustre: Skipped 2 previous similar messages [11331.593975] LustreError: Skipped 1 previous similar message [11336.674921] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11336.681694] Lustre: Skipped 2 previous similar messages [11341.797381] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11341.803220] Lustre: Skipped 1 previous similar message [11345.376132] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11345.468505] Lustre: server umount lustre-OST0000 complete [11352.034601] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11352.046311] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11352.050408] Lustre: Skipped 1 previous similar message [11362.272759] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11362.397871] Lustre: server umount lustre-MDT0000 complete [11364.838796] LustreError: 288782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773618504 with bad export cookie 862918855988459476 [11364.847997] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11364.850445] LustreError: 288782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [11365.176473] Lustre: server umount lustre-MDT0001 complete [11370.848781] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 19:48:29 (1773618509) [11371.840767] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [11373.149542] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 19:48:31 (1773618511) [11375.110384] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [11393.383649] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11393.579377] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11393.598457] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11393.719142] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11393.725044] Lustre: Skipped 1 previous similar message [11396.552246] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11402.079821] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11402.185960] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11402.204631] Lustre: 292474:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11405.061904] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11408.372876] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11410.717996] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11427.812679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11427.939274] Lustre: Denying initial registration attempt from nid 192.168.201.105@tcp, specified as failover [11427.945120] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [11427.952385] LustreError: 294085:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -99 [11427.956425] LustreError: 294085:0:(tgt_mount.c:2064:server_put_super()) no obd lustre-OST0000 [11427.960333] LustreError: 294085:0:(tgt_mount.c:115:server_deregister_mount()) lustre-OST0000 not registered [11428.011759] Lustre: server umount lustre-OST0000 complete [11428.015192] LustreError: 294085:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -99 [11432.929163] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11432.933978] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11432.941342] Lustre: Skipped 1 previous similar message [11432.945078] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11432.952537] Lustre: Skipped 4 previous similar messages [11437.681523] Lustre: server umount lustre-MDT0000 complete [11439.703815] LustreError: 292466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773618579 with bad export cookie 862918855988460358 [11439.709262] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11439.716447] LustreError: 292466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11439.881080] Lustre: server umount lustre-MDT0001 complete [11446.277823] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [11448.110313] Key type lgssc unregistered [11448.305693] LNet: 295189:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11448.310579] LNetError: 295189:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11448.320409] LNet: Removed LNI 192.168.201.105@tcp [11448.875614] Key type .llcrypt unregistered [11448.879306] Key type ._llcrypt unregistered [11455.481519] Key type ._llcrypt registered [11455.483600] Key type .llcrypt registered [11455.544397] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [11456.052326] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11456.107802] alg: No test for adler32 (adler32-zlib) [11457.039151] Lustre: Lustre: Build Version: 2.17.51_1_gce01c51 [11457.180712] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [11458.792184] Key type lgssc registered [11459.436095] Lustre: Echo OBD driver; http://www.lustre.org/ [11503.239129] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [11507.848077] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11507.861844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11509.053944] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11509.072843] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11509.179258] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11511.225353] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11515.584592] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11515.667047] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11515.685041] Lustre: 297903:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11515.827885] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11517.926547] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11520.786992] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11522.790350] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11540.198025] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11540.307372] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11540.413556] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11543.205517] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11545.979213] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11547.136973] Lustre: server umount lustre-OST0000 complete [11551.713752] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11551.717561] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11551.723032] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11553.250039] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11553.255547] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11555.188122] Lustre: server umount lustre-MDT0000 complete [11556.978866] LustreError: 297894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773618697 with bad export cookie 7343129236201368087 [11556.982930] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11556.990065] LustreError: 297894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11557.147178] Lustre: server umount lustre-MDT0001 complete [11562.503384] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [11563.936747] Key type lgssc unregistered [11564.109431] LNet: 301115:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11564.113469] LNetError: 301115:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11564.122353] LNet: Removed LNI 192.168.201.105@tcp [11564.541051] Key type .llcrypt unregistered [11564.543097] Key type ._llcrypt unregistered [11577.560026] Key type ._llcrypt registered [11577.561805] Key type .llcrypt registered [11577.621995] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [11578.200103] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11578.221863] alg: No test for adler32 (adler32-zlib) [11579.140248] Lustre: Lustre: Build Version: 2.17.51_1_gce01c51 [11579.260141] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [11580.864205] Key type lgssc registered [11581.544919] Lustre: Echo OBD driver; http://www.lustre.org/ [11586.471861] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11586.489941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11587.823167] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11589.987584] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11594.858604] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11595.087313] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11597.227512] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11600.127589] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11601.933382] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11607.463058] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11607.666590] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11610.845263] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11613.975612] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11625.443041] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11625.445550] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11625.449675] Lustre: Skipped 2 previous similar messages [11625.452780] Lustre: Skipped 1 previous similar message [11626.581679] Lustre: server umount lustre-MDT0000 complete [11628.685792] LustreError: 302474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773618768 with bad export cookie 481244621835347058 [11628.690283] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11628.692239] LustreError: 302474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11630.561213] LustreError: 302488:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11630.561581] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11630.572882] LustreError: 302488:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [11630.587022] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11630.594372] Lustre: Skipped 1 previous similar message [11633.569141] LustreError: 304105:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11633.571110] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11633.579258] LustreError: 304105:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [11633.594244] Lustre: Skipped 1 previous similar message [11634.945033] Lustre: server umount lustre-MDT0001 complete [11638.317561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [11642.509694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11653.092714] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x6adb96f0e904048 [11653.098617] Lustre: MGC192.168.201.105@tcp: Connection restored to 0@lo (at 0@lo) [11653.268910] LustreError: 304095:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11653.314837] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11653.352154] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [11655.473205] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11658.403044] LustreError: 305746:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11658.410591] LustreError: 305746:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [11660.176510] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11660.399850] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11662.459591] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11663.463621] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [11665.277959] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11668.276629] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [11670.100224] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11672.203794] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 1 sec [11675.617778] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11675.621400] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11675.630507] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11679.206481] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11679.207111] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11680.706173] Lustre: server umount lustre-OST0000 complete [11684.324494] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11684.325056] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11684.332551] Lustre: Skipped 1 previous similar message [11684.335589] Lustre: Skipped 2 previous similar messages [11688.698099] Lustre: server umount lustre-MDT0000 complete [11689.444090] LustreError: 305746:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11689.454297] LustreError: 305746:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [11690.692411] LustreError: 302472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773618830 with bad export cookie 481244621835354184 [11690.696394] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11690.700136] LustreError: 302472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11690.865087] Lustre: server umount lustre-MDT0001 complete [11697.198250] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [11698.670471] Key type lgssc unregistered [11698.854106] LNet: 308181:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11698.862555] LNetError: 308181:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11698.878359] LNet: Removed LNI 192.168.201.105@tcp [11699.367879] Key type .llcrypt unregistered [11699.369837] Key type ._llcrypt unregistered [11713.292361] Key type ._llcrypt registered [11713.293760] Key type .llcrypt registered [11713.357451] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [11713.951384] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11713.969962] alg: No test for adler32 (adler32-zlib) [11714.931210] Lustre: Lustre: Build Version: 2.17.51_1_gce01c51 [11715.084639] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [11716.712330] Key type lgssc registered [11717.411706] Lustre: Echo OBD driver; http://www.lustre.org/ [11722.010656] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11722.033020] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11723.323523] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11725.710442] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11731.177292] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11731.422247] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11734.077275] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11737.845782] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11740.391564] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11742.515555] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11742.520052] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11742.523566] Lustre: Skipped 1 previous similar message [11742.536260] Lustre: Skipped 1 previous similar message [11747.810693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11747.814444] Lustre: Skipped 1 previous similar message [11748.788185] Lustre: server umount lustre-MDT0000 complete [11751.458918] LustreError: 310950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773618891 with bad export cookie 11054025214257280354 [11751.467776] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11751.468253] LustreError: 310950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11751.679759] Lustre: server umount lustre-MDT0001 complete [11761.283896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11761.468117] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11761.492248] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11761.617283] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11763.848904] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11768.796201] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11768.894536] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11768.921371] Lustre: 311920:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11769.068524] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11771.222855] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11774.848552] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11777.282307] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11783.849352] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11783.993183] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [11783.999220] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [11784.007657] LustreError: 313577:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [11784.013410] LustreError: 313577:0:(tgt_mount.c:2064:server_put_super()) no obd lustre-OST0000 [11784.018314] LustreError: 313577:0:(tgt_mount.c:115:server_deregister_mount()) lustre-OST0000 not registered [11784.069815] Lustre: server umount lustre-OST0000 complete [11784.073854] LustreError: 313577:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -2 [11791.017410] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11791.153404] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11791.273863] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11794.306400] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11797.578373] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11801.168211] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [11802.460938] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11802.513518] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [11802.558549] Lustre: lustre-OST0001: new disk, initializing [11802.561086] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11802.601228] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11806.319384] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11809.462163] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [11810.863871] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11810.871033] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11810.938360] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11815.911535] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11815.912900] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11815.916231] LustreError: Skipped 1 previous similar message [11815.917873] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11815.927182] Lustre: Skipped 1 previous similar message [11817.250150] Lustre: server umount lustre-OST0001 complete [11821.030224] LustreError: 314043:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11821.031768] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11821.033032] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11821.033037] Lustre: Skipped 1 previous similar message [11821.043265] LustreError: 314043:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [11821.050241] Lustre: Skipped 1 previous similar message [11825.475430] Lustre: server umount lustre-OST0000 complete [11830.755166] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11830.762105] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11830.769540] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11830.774368] Lustre: Skipped 1 previous similar message [11833.977765] Lustre: server umount lustre-MDT0000 complete [11835.993455] LustreError: 311912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773618976 with bad export cookie 11054025214257280893 [11835.997114] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11835.999068] LustreError: 311912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11836.171434] Lustre: server umount lustre-MDT0001 complete [11841.523580] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [11842.950701] Key type lgssc unregistered [11843.125376] LNet: 316829:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11843.129197] LNetError: 316829:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11843.140056] LNet: Removed LNI 192.168.201.105@tcp [11843.593506] Key type .llcrypt unregistered [11843.595974] Key type ._llcrypt unregistered [11889.049247] Key type ._llcrypt registered [11889.050929] Key type .llcrypt registered [11889.118238] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [11897.634063] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [11898.184151] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11898.253385] alg: No test for adler32 (adler32-zlib) [11899.178185] Lustre: Lustre: Build Version: 2.17.51_1_gce01c51 [11899.297901] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [11900.904376] Key type lgssc registered [11901.522906] Lustre: Echo OBD driver; http://www.lustre.org/ [11920.324721] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11920.334448] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11921.455485] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11921.471379] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11921.515378] Lustre: lustre-MDT0000: new disk, initializing [11921.549846] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11921.560285] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11923.345955] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11928.648245] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11928.711822] Lustre: 321702:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11928.746042] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11928.749016] Lustre: Skipped 1 previous similar message [11928.794109] Lustre: lustre-MDT0001: new disk, initializing [11928.841723] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11928.865026] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11928.870586] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11930.820725] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11935.243821] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11937.277496] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11940.928690] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11941.091389] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [11941.095409] Lustre: lustre-OST0000: new disk, initializing [11941.100623] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11941.155576] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11944.469983] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11946.495287] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11946.503169] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11946.522962] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [11948.950880] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11951.097626] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11951.212628] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11953.184971] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11953.300525] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11956.705189] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11956.710159] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11956.718584] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11959.268219] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11959.268917] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11959.275068] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11960.489536] Lustre: server umount lustre-OST0000 complete [11964.386266] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11964.387597] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11964.393290] Lustre: Skipped 1 previous similar message [11964.398440] Lustre: Skipped 1 previous similar message [11968.659996] Lustre: server umount lustre-MDT0000 complete [11971.796252] LustreError: 321696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773619111 with bad export cookie 17014117033393994088 [11971.797739] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11971.800820] LustreError: 321696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11971.992718] Lustre: server umount lustre-MDT0001 complete [11976.437224] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 19:58:35 (1773619115) [11980.210628] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 19:58:38 (1773619118) [11984.625996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11984.856108] LustreError: 326052:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11984.864671] LustreError: 326052:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [11984.887621] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11986.667389] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11989.986323] LustreError: 326053:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11990.647657] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11990.847569] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11992.843221] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11995.587424] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11997.389102] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12000.770261] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12000.965770] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12003.803296] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [12006.566781] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12021.084822] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [12022.035511] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [12022.815762] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [12023.793075] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [12024.782560] Lustre: Failing over lustre-MDT0000 [12025.826894] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12025.828248] LustreError: 327117:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12025.831252] Lustre: Skipped 2 previous similar messages [12025.837468] LustreError: 327117:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [12027.004578] Lustre: server umount lustre-MDT0000 complete [12028.548764] LustreError: 327117:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12033.654965] LustreError: 328206:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12033.667375] LustreError: 328206:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [12041.184128] Lustre: 319931:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773619165/real 1773619165] req@000000000c8aa9c4 x1859774357496320/t0(0) o400->MGC192.168.201.105@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1773619181 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12041.202257] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12042.033268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12043.900500] LustreError: 326047:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12043.924540] LustreError: 326047:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 7 previous similar messages [12051.431096] LustreError: 319927:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000e9c96832 x1859774357501696/t0(0) o250->MGC192.168.201.105@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12051.673341] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12051.709604] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [12052.631350] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [12054.148948] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [12057.069615] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [12057.095096] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [12057.131329] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:35 to 0x280000400:65) [12059.117276] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12060.050311] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12062.431192] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [12063.433160] Lustre: Failing over lustre-MDT0000 [12063.668152] Lustre: server umount lustre-MDT0000 complete [12067.298674] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12067.300121] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12067.300874] LustreError: 326049:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12067.300880] LustreError: 326049:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 7 previous similar messages [12067.327443] Lustre: Skipped 2 previous similar messages [12078.247359] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12078.321722] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12078.539103] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [12080.522863] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [12082.806681] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [12083.688307] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12083.690984] Lustre: Skipped 2 previous similar messages [12083.702950] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [12083.722564] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:35 to 0x280000400:97) [12085.415123] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12086.133089] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12093.924312] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12093.924619] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12093.936173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12093.939535] Lustre: Skipped 3 previous similar messages [12095.235393] Lustre: server umount lustre-MDT0000 complete [12097.086934] LustreError: 326033:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773619237 with bad export cookie 17014117033393996797 [12097.089509] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12097.095327] LustreError: 326033:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12097.270328] Lustre: server umount lustre-MDT0001 complete [12109.755996] Lustre: server umount lustre-OST0000 complete [12117.437536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12117.713164] LustreError: 331851:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12117.724876] LustreError: 331851:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 11 previous similar messages [12117.759436] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12117.764494] Lustre: Skipped 1 previous similar message [12119.661816] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [12124.399380] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12126.739058] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [12129.490699] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12131.510238] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12135.223312] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12138.166862] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [12140.989584] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12142.633411] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:35 to 0x280000400:129) [12154.852355] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12154.856599] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12154.857939] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12154.864891] Lustre: Skipped 3 previous similar messages [12159.142824] Lustre: server umount lustre-OST0000 complete [12175.329902] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12175.330600] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 4. Is it stuck? [12175.334075] Lustre: Skipped 6 previous similar messages [12175.413107] Lustre: server umount lustre-MDT0000 complete [12177.038274] LustreError: 331831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773619317 with bad export cookie 17014117033393997518 [12177.038904] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12177.043280] LustreError: 331831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12177.198755] Lustre: server umount lustre-MDT0001 complete [12181.911126] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [12183.194384] Key type lgssc unregistered [12183.350431] LNet: 335061:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12183.355384] LNetError: 335061:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12183.366789] LNet: Removed LNI 192.168.201.105@tcp [12183.753460] Key type .llcrypt unregistered [12183.755584] Key type ._llcrypt unregistered [12196.404793] Key type ._llcrypt registered [12196.406693] Key type .llcrypt registered [12196.464264] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [12196.955745] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12196.991869] alg: No test for adler32 (adler32-zlib) [12197.912742] Lustre: Lustre: Build Version: 2.17.51_1_gce01c51 [12198.021526] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [12199.632442] Key type lgssc registered [12200.135076] Lustre: Echo OBD driver; http://www.lustre.org/ [12203.781441] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12203.791988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12205.066428] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12206.997411] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [12211.132280] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12211.316695] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12213.152077] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [12215.771309] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12217.722285] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12220.999704] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12221.189671] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12223.853483] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [12226.460315] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12230.442185] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:35 to 0x280000400:161) [12240.868925] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12240.870461] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12240.877710] Lustre: Skipped 2 previous similar messages [12240.883440] Lustre: Skipped 2 previous similar messages [12243.428789] Lustre: server umount lustre-MDT0000 complete [12245.095370] LustreError: 336519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773619385 with bad export cookie 16733040126503576600 [12245.099173] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12245.101975] LustreError: 336519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12245.246347] Lustre: server umount lustre-MDT0001 complete [12257.390882] Lustre: server umount lustre-OST0000 complete [12263.754806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12263.994042] LustreError: 339912:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12264.004553] LustreError: 339912:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [12264.027238] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12265.921118] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [12269.025085] LustreError: 339912:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12269.033719] LustreError: 339912:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [12269.713164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12269.887942] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12271.646488] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [12273.962969] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12275.533032] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12278.359063] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12278.546338] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12281.164886] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [12283.482161] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12289.832050] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [12305.381446] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12305.382878] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12305.389131] Lustre: Skipped 1 previous similar message [12308.009295] Lustre: server umount lustre-MDT0000 complete [12309.677324] LustreError: 339892:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773619449 with bad export cookie 16733040126503577699 [12309.680831] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12309.683252] LustreError: 339892:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12309.828159] Lustre: server umount lustre-MDT0001 complete [12321.905473] Lustre: server umount lustre-OST0000 complete [12328.706986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12328.902495] LustreError: 343506:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12328.930044] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12330.560169] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [12334.049666] LustreError: 343507:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12334.819436] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12336.772499] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [12339.202940] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12340.779935] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12343.688692] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12346.389489] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [12348.760789] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12353.069868] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:225) [12356.069087] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12356.072574] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12356.080318] Lustre: Skipped 1 previous similar message [12356.083601] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12356.086903] Lustre: Skipped 2 previous similar messages [12358.113097] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12358.119459] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12358.119616] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12358.127314] Lustre: Skipped 1 previous similar message [12361.763896] Lustre: server umount lustre-OST0000 complete [12365.793541] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12365.798084] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12365.804976] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12369.574542] Lustre: server umount lustre-MDT0000 complete [12371.251194] LustreError: 344225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773619511 with bad export cookie 16733040126503578707 [12371.252526] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12371.260726] LustreError: 344225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12371.410128] Lustre: server umount lustre-MDT0001 complete [12376.716353] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [12378.059273] Key type lgssc unregistered [12378.222393] LNet: 346689:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12378.226399] LNetError: 346689:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12378.233522] LNet: Removed LNI 192.168.201.105@tcp [12378.675250] Key type .llcrypt unregistered [12378.677175] Key type ._llcrypt unregistered [12391.516631] Key type ._llcrypt registered [12391.518936] Key type .llcrypt registered [12391.574580] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [12392.074657] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12392.129386] alg: No test for adler32 (adler32-zlib) [12393.014120] Lustre: Lustre: Build Version: 2.17.51_1_gce01c51 [12393.134154] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [12394.736166] Key type lgssc registered [12395.209707] Lustre: Echo OBD driver; http://www.lustre.org/ [12398.776871] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12398.784826] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12398.789951] LustreError: 348088:0:(osd_handler.c:8547:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [12398.800929] LustreError: 348088:0:(obd_config.c:880:class_setup()) setup lustre-MDT0000-osd failed (-22) [12398.804975] LustreError: 348088:0:(obd_mount.c:217:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [12398.809699] LustreError: 348088:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [12398.815055] LustreError: 348088:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -22 [12402.190634] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [12402.194476] LustreError: 348486:0:(osd_handler.c:8547:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [12402.204679] LustreError: 348486:0:(obd_config.c:880:class_setup()) setup lustre-OST0000-osd failed (-22) [12402.208801] LustreError: 348486:0:(obd_mount.c:217:lustre_start_simple()) lustre-OST0000-osd setup error -22 [12402.211624] LustreError: 348486:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [12402.217284] LustreError: 348486:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -22 [12410.901505] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [12412.135043] Key type lgssc unregistered [12412.275326] LNet: 349470:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12412.279193] LNetError: 349470:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12412.292883] LNet: Removed LNI 192.168.201.105@tcp [12412.656460] Key type .llcrypt unregistered [12412.658088] Key type ._llcrypt unregistered [12419.581103] Key type ._llcrypt registered [12419.583364] Key type .llcrypt registered [12419.636654] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [12426.983527] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [12427.437803] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12427.499282] alg: No test for adler32 (adler32-zlib) [12428.363940] Lustre: Lustre: Build Version: 2.17.51_1_gce01c51 [12428.464066] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [12430.056161] Key type lgssc registered [12430.556959] Lustre: Echo OBD driver; http://www.lustre.org/ [12445.605963] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12445.616932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12446.750039] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12446.771797] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12446.821152] Lustre: lustre-MDT0000: new disk, initializing [12446.856161] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12446.865118] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12448.641349] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [12454.195877] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12454.262630] Lustre: 352566:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12454.293475] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12454.296514] Lustre: Skipped 1 previous similar message [12454.343609] Lustre: lustre-MDT0001: new disk, initializing [12454.408480] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12454.429447] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12454.438682] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12456.359294] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [12460.548961] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12462.431251] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12465.720335] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12465.882632] Lustre: lustre-OST0000: new disk, initializing [12465.886252] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12465.930681] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12468.736679] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [12469.266557] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12469.272198] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12469.332258] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12472.894546] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12474.848399] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12474.955344] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12476.957293] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12477.084472] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12479.462206] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12479.462250] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12479.471426] LustreError: Skipped 1 previous similar message [12479.481554] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12479.487085] Lustre: Skipped 1 previous similar message [12484.578691] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12484.581728] Lustre: Skipped 1 previous similar message [12489.702131] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12489.706417] Lustre: Skipped 1 previous similar message [12492.256149] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12492.336173] Lustre: server umount lustre-OST0000 complete [12494.819820] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12494.825806] Lustre: Skipped 1 previous similar message [12494.828277] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12494.832806] Lustre: Skipped 1 previous similar message [12499.937971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12499.941971] Lustre: Skipped 1 previous similar message [12500.217546] Lustre: server umount lustre-MDT0000 complete [12502.006955] LustreError: 354307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773619642 with bad export cookie 126723504099276318 [12502.012056] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12502.014563] LustreError: 354307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12502.156097] Lustre: server umount lustre-MDT0001 complete [12506.330107] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [12507.183496] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 20:07:25 (1773619645) [12510.319806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12510.594714] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12512.585433] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [12515.811799] LustreError: 356619:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12515.820459] LustreError: 356619:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [12516.650951] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12516.832685] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12518.801844] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [12521.533529] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12523.251422] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12526.295532] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12526.444523] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12528.774276] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [12531.182632] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12534.271398] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12534.352275] Lustre: lustre-OST0001: new disk, initializing [12534.354056] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [12536.738914] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [12539.408989] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12539.413820] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12539.457801] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12540.331544] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [12544.482187] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12544.482273] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12544.485485] LustreError: Skipped 1 previous similar message [12544.486032] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [12544.491898] Lustre: Skipped 2 previous similar messages [12548.560923] Lustre: server umount lustre-OST0001 complete [12551.653625] LustreError: 359657:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12551.661804] LustreError: 359657:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [12556.772455] LustreError: 359657:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12556.777635] LustreError: 359657:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [12561.893752] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12561.895847] LustreError: 359657:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12561.897522] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12561.904043] LustreError: 359657:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [12561.911840] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12561.918217] Lustre: Skipped 5 previous similar messages [12567.013829] LustreError: 358214:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12567.023151] LustreError: 358214:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [12567.189432] Lustre: server umount lustre-OST0000 complete [12572.129964] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12572.138748] Lustre: Skipped 2 previous similar messages [12574.885204] Lustre: server umount lustre-MDT0000 complete [12576.394854] LustreError: 356598:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773619716 with bad export cookie 126723504099277200 [12576.399293] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12576.400660] LustreError: 356598:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12582.592531] Lustre: server umount lustre-MDT0001 complete [12587.059955] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [12588.246582] Key type lgssc unregistered [12588.409716] LNet: 360974:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12588.414314] LNetError: 360974:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12588.427388] LNet: Removed LNI 192.168.201.105@tcp [12588.833164] Key type .llcrypt unregistered [12588.834878] Key type ._llcrypt unregistered [12609.491536] Key type ._llcrypt registered [12609.492521] Key type .llcrypt registered [12609.528831] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [12610.003388] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12610.036380] alg: No test for adler32 (adler32-zlib) [12610.924454] Lustre: Lustre: Build Version: 2.17.51_1_gce01c51 [12611.042741] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [12612.640148] Key type lgssc registered [12613.086936] Lustre: Echo OBD driver; http://www.lustre.org/ [12616.127335] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12616.135220] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12617.241862] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12617.252268] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12617.321244] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12618.806093] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [12622.256981] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12622.307189] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12622.323188] Lustre: 363732:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12622.451719] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12623.975741] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [12626.077713] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12627.516167] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12632.546322] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12632.546388] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12632.547190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12632.555821] Lustre: Skipped 1 previous similar message [12634.400466] Lustre: server umount lustre-MDT0000 complete [12635.698205] LustreError: 364456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773619775 with bad export cookie 12169088802125901539 [12635.700881] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12635.701737] LustreError: 364456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12635.835359] Lustre: server umount lustre-MDT0001 complete [12639.318726] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 20:09:38 (1773619778) [12641.642472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12641.818146] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12643.128792] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [12646.191540] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12646.318972] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12647.627900] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [12649.607094] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12650.943984] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12653.449873] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12653.552646] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [12653.627796] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12655.960762] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [12658.216380] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12662.152517] Lustre: Permanently deactivating lustre-OST0000 [12662.158644] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [12665.716245] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12666.877549] LustreError: 368610:0:(mgs_llog.c:1942:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [12666.883458] LustreError: 368610:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [12666.887500] LustreError: 368610:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12667.580235] LustreError: 368658:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [12667.583233] LustreError: 368658:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12674.470529] Lustre: server umount lustre-OST0000 complete [12675.615113] LustreError: 368907:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [12675.619437] LustreError: 368907:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12677.090762] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12677.091097] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12677.091603] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12677.091607] Lustre: Skipped 1 previous similar message [12677.104418] Lustre: Skipped 1 previous similar message [12682.209435] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12682.212022] Lustre: Skipped 2 previous similar messages [12686.330255] Lustre: server umount lustre-MDT0000 complete [12687.329258] LustreError: 366015:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12687.337283] LustreError: 366015:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [12687.619384] LustreError: 367619:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773619827 with bad export cookie 12169088802125902106 [12687.623356] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12687.624121] LustreError: 367619:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12697.106874] Lustre: server umount lustre-MDT0001 complete [12700.291649] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12701.833948] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [12702.847227] LustreError: 370048:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [12705.301993] Lustre: 370387:0:(mgs_llog.c:1709:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.105@tcp [12706.044891] Lustre: server umount MGS complete [12710.386480] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12710.565561] LustreError: 371110:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12710.575559] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12710.578814] Lustre: Skipped 1 previous similar message [12710.604330] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12712.321543] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [12715.813700] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12715.930260] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12717.527234] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [12719.694911] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12721.223641] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12721.998346] Lustre: Permanently reactivating lustre-OST0000 [12722.005103] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12722.007986] Lustre: Skipped 2 previous similar messages [12745.696165] LustreError: 372616:0:(osp_dev.c:694:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12749.503418] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12749.665429] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12749.668774] Lustre: Skipped 1 previous similar message [12752.333840] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [12754.637558] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12771.299847] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12771.301729] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12771.301889] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12771.305728] Lustre: Skipped 1 previous similar message [12771.311779] Lustre: Skipped 1 previous similar message [12773.226850] Lustre: server umount lustre-OST0000 complete [12776.417829] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12776.418511] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12776.422339] Lustre: Skipped 1 previous similar message [12786.657335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12786.659541] Lustre: Skipped 4 previous similar messages [12789.216171] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12789.279150] Lustre: server umount lustre-MDT0000 complete [12790.684382] LustreError: 371827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773619930 with bad export cookie 12169088802125903142 [12790.687555] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12790.688704] LustreError: 371827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12790.817757] Lustre: server umount lustre-MDT0001 complete [12795.032610] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [12796.157182] Key type lgssc unregistered [12796.292387] LNet: 375333:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12796.296162] LNetError: 375333:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12796.304522] LNet: Removed LNI 192.168.201.105@tcp [12796.637481] Key type .llcrypt unregistered [12796.638631] Key type ._llcrypt unregistered [12803.853862] Key type ._llcrypt registered [12803.857200] Key type .llcrypt registered [12803.905493] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [12810.696030] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [12811.316772] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12811.330582] alg: No test for adler32 (adler32-zlib) [12812.207710] Lustre: Lustre: Build Version: 2.17.51_1_gce01c51 [12812.313135] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [12813.896264] Key type lgssc registered [12814.378705] Lustre: Echo OBD driver; http://www.lustre.org/ [12827.365494] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12827.370951] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12828.474965] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12828.487800] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12828.531607] Lustre: lustre-MDT0000: new disk, initializing [12828.562775] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12828.569557] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12830.223771] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [12835.008794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12835.067130] Lustre: 378427:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12835.102443] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12835.105724] Lustre: Skipped 1 previous similar message [12835.154709] Lustre: lustre-MDT0001: new disk, initializing [12835.192255] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12835.206922] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12835.212644] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12836.738754] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [12840.125247] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12841.529707] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12844.242416] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12844.371642] Lustre: lustre-OST0000: new disk, initializing [12844.374085] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12844.407259] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12846.563110] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [12849.152523] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12849.156843] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12849.195118] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12849.907643] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12851.483668] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12851.581484] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12853.061037] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12853.143430] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12854.242378] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12854.243612] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12854.244821] LustreError: Skipped 1 previous similar message [12854.248504] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12854.249765] Lustre: Skipped 1 previous similar message [12855.970463] Lustre: server umount lustre-OST0000 complete [12859.362316] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12859.363786] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12859.366948] Lustre: Skipped 1 previous similar message [12859.369150] Lustre: Skipped 1 previous similar message [12863.659589] Lustre: server umount lustre-MDT0000 complete [12865.049646] LustreError: 378422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773620005 with bad export cookie 11284414260462376903 [12865.052868] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12865.053907] LustreError: 378422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12865.192291] Lustre: server umount lustre-MDT0001 complete [12868.274853] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12868.856787] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 20:13:27 (1773620007) [12871.259346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12871.431818] LustreError: 382480:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12871.437848] LustreError: 382480:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [12871.459024] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12872.985290] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [12876.182770] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12876.337899] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12877.758890] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [12879.818848] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12881.245522] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12883.552961] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12885.601685] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [12887.482784] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12888.239577] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12899.004221] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12902.371603] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12902.372152] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12902.372711] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12902.372716] Lustre: Skipped 1 previous similar message [12902.375324] Lustre: Skipped 1 previous similar message [12902.377318] LustreError: Skipped 1 previous similar message [12906.711260] Lustre: server umount lustre-OST0000 complete [12912.097158] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12912.100494] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12912.105601] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12912.108341] Lustre: Skipped 1 previous similar message [12914.267066] Lustre: server umount lustre-MDT0000 complete [12915.536139] LustreError: 382461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773620055 with bad export cookie 11284414260462377785 [12915.538648] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12915.540699] LustreError: 382461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12915.664983] Lustre: server umount lustre-MDT0001 complete [12919.635398] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [12920.753409] Key type lgssc unregistered [12920.873403] LNet: 385872:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12920.876083] LNetError: 385872:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12920.882343] LNet: Removed LNI 192.168.201.105@tcp [12921.195715] Key type .llcrypt unregistered [12921.196912] Key type ._llcrypt unregistered [12930.064661] Key type ._llcrypt registered [12930.065789] Key type .llcrypt registered [12930.108189] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [12930.227340] Key type .llcrypt unregistered [12930.228398] Key type ._llcrypt unregistered [12937.571600] Key type ._llcrypt registered [12937.572795] Key type .llcrypt registered [12937.617429] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [12938.165091] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12938.174287] alg: No test for adler32 (adler32-zlib) [12939.039267] Lustre: Lustre: Build Version: 2.17.51_1_gce01c51 [12939.125888] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [12940.712239] Key type lgssc registered [12941.117376] Lustre: Echo OBD driver; http://www.lustre.org/ [12944.067499] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12944.074713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12945.247351] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12946.592397] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [12949.607848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12949.734090] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12951.847371] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [12953.917829] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12955.105760] LustreError: 388027:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12955.113438] LustreError: 388027:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 4 previous similar messages [12956.130309] LustreError: 388026:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12956.135041] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12956.139620] LustreError: 388026:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [12956.549129] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12956.691796] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12958.227816] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [12966.369701] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12966.372567] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12966.377309] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12966.887807] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12966.889272] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12966.897189] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12972.006918] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12972.010762] Lustre: Skipped 1 previous similar message [12972.072440] Lustre: server umount lustre-OST0000 complete [12976.098364] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [12976.101424] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12976.105464] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12979.810138] Lustre: server umount lustre-MDT0000 complete [12981.262650] LustreError: 388006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773620121 with bad export cookie 2954045331192890041 [12981.266983] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12981.267477] LustreError: 388006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12981.416293] Lustre: server umount lustre-MDT0001 complete [12985.675596] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [12986.921024] Key type lgssc unregistered [12987.064404] LNet: 391010:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12987.070220] LNetError: 391010:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12987.080507] LNet: Removed LNI 192.168.201.105@tcp [12987.447829] Key type .llcrypt unregistered [12987.449204] Key type ._llcrypt unregistered [12997.805694] Key type ._llcrypt registered [12997.807041] Key type .llcrypt registered [12997.847152] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [12998.318644] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12998.336717] alg: No test for adler32 (adler32-zlib) [12999.217057] Lustre: Lustre: Build Version: 2.17.51_1_gce01c51 [12999.324762] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [13000.912305] Key type lgssc registered [13001.339972] Lustre: Echo OBD driver; http://www.lustre.org/ [13004.618701] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13004.630716] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13020.128388] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [13025.249778] LustreError: 392361:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.105@tcp: failed processing log, type 4: rc = -110 [13051.872414] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13054.083184] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [13056.198068] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13058.860882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13059.075876] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13059.105476] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [13060.621471] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [13064.161539] LustreError: 392387:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13064.167955] LustreError: 392387:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [13064.268493] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13064.458325] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13066.249757] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [13068.536939] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13070.100767] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13074.401207] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13074.405258] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13074.411454] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13074.914178] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13074.917187] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13074.921872] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13076.961958] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13076.964983] Lustre: Skipped 1 previous similar message [13078.752539] Lustre: server umount lustre-OST0000 complete [13082.082403] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13082.083429] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13082.089088] Lustre: Skipped 1 previous similar message [13086.495236] Lustre: server umount lustre-MDT0000 complete [13087.201034] LustreError: 393286:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13087.208782] LustreError: 393286:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [13088.041384] LustreError: 392367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773620228 with bad export cookie 16758140640298758273 [13088.044655] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13088.046325] LustreError: 392367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13088.203040] Lustre: server umount lustre-MDT0001 complete [13093.018201] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [13094.324601] Key type lgssc unregistered [13094.475363] LNet: 395575:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13094.479936] LNetError: 395575:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13094.492420] LNet: Removed LNI 192.168.201.105@tcp [13094.901441] Key type .llcrypt unregistered [13094.904165] Key type ._llcrypt unregistered [13107.152544] Key type ._llcrypt registered [13107.153874] Key type .llcrypt registered [13107.203445] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [13107.615513] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13107.677932] alg: No test for adler32 (adler32-zlib) [13108.561174] Lustre: Lustre: Build Version: 2.17.51_1_gce01c51 [13108.688591] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [13110.296399] Key type lgssc registered [13110.836930] Lustre: Echo OBD driver; http://www.lustre.org/ [13114.465994] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13114.476164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13115.727685] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13117.457944] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [13121.280607] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13121.446575] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13123.203893] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [13125.550081] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13127.168898] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13130.398205] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13130.589546] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13133.254113] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [13135.668175] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13140.841062] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [13141.986035] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13141.988058] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13141.993961] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13146.081763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13146.083625] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13146.091325] Lustre: Skipped 1 previous similar message [13148.133980] Lustre: server umount lustre-MDT0000 complete [13149.109631] LustreError: 396933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.5@tcp arrived at 1773620289 with bad export cookie 6614828825915649521 [13149.115801] LustreError: 396933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13149.954468] LustreError: 396932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773620290 with bad export cookie 6614828825915648898 [13149.960977] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13151.201279] LustreError: 397675:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13151.201813] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13151.206762] LustreError: 397675:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [13151.213499] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13151.218581] Lustre: Skipped 3 previous similar messages [13154.208967] LustreError: 396951:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13154.209291] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13154.215526] LustreError: 396951:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [13156.068298] Lustre: server umount lustre-OST0000 complete [13159.046353] Lustre: server umount lustre-MDT0001 complete [13163.492619] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [13164.654738] Key type lgssc unregistered [13164.799702] LNet: 400282:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13164.802727] LNetError: 400282:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13164.815302] LNet: Removed LNI 192.168.201.105@tcp [13165.144814] Key type .llcrypt unregistered [13165.146403] Key type ._llcrypt unregistered [13175.620876] Key type ._llcrypt registered [13175.622135] Key type .llcrypt registered [13175.666293] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [13176.089859] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13176.139398] alg: No test for adler32 (adler32-zlib) [13177.022873] Lustre: Lustre: Build Version: 2.17.51_1_gce01c51 [13177.129752] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [13178.728210] Key type lgssc registered [13179.180092] Lustre: Echo OBD driver; http://www.lustre.org/ [13182.361695] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13182.369895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13183.560659] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13185.082441] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [13188.346884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13188.486358] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13189.910881] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [13192.002737] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13193.373570] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13195.936557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13196.085058] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13198.253635] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [13200.295734] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13208.359451] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [13213.669549] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13213.675676] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13213.677738] Lustre: Skipped 1 previous similar message [13215.868584] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.5@tcp (stopping) [13217.877895] Lustre: server umount lustre-MDT0001 complete [13218.788429] LustreError: 402921:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13218.795135] LustreError: 402921:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [13223.905806] LustreError: 401663:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13223.906513] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13223.906531] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13223.906537] Lustre: Skipped 1 previous similar message [13223.907138] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13225.879333] Lustre: server umount lustre-OST0000 complete [13227.642847] Lustre: server umount lustre-MDT0000 complete [13233.235370] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [13234.541944] Key type lgssc unregistered [13234.675348] LNet: 405013:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13234.679374] LNetError: 405013:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13234.689573] LNet: Removed LNI 192.168.201.105@tcp [13235.079107] Key type .llcrypt unregistered [13235.080314] Key type ._llcrypt unregistered [13244.816087] Key type ._llcrypt registered [13244.817414] Key type .llcrypt registered [13244.875893] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [13251.686520] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [13252.157379] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13252.165925] alg: No test for adler32 (adler32-zlib) [13253.058193] Lustre: Lustre: Build Version: 2.17.51_1_gce01c51 [13253.167482] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [13254.752143] Key type lgssc registered [13255.163641] Lustre: Echo OBD driver; http://www.lustre.org/ [13268.579081] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [13274.290253] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13274.297033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13275.401358] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13275.416740] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13275.460764] Lustre: lustre-MDT0000: new disk, initializing [13275.486730] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13275.495275] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13276.971193] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [13282.786391] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13282.899118] Lustre: lustre-OST0000: new disk, initializing [13282.901264] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13282.903366] Lustre: Skipped 1 previous similar message [13282.927995] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13284.032326] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13284.036551] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13284.046642] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13285.082970] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [13288.299172] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13294.186882] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13294.232753] Lustre: 408950:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13294.257610] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13294.293624] Lustre: lustre-MDT0001: new disk, initializing [13294.339193] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13294.354639] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13294.358544] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13295.874428] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [13299.961421] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13300.049930] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13301.602955] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13301.694680] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13305.314273] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13305.314976] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13305.318863] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13305.323308] Lustre: Skipped 1 previous similar message [13305.329322] Lustre: Skipped 1 previous similar message [13305.979658] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.5@tcp (stopping) [13309.729685] Lustre: server umount lustre-MDT0001 complete [13315.554079] LustreError: 408954:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13315.558557] LustreError: 408954:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 5 previous similar messages [13320.673988] LustreError: 408955:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13320.682908] LustreError: 408955:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [13325.281029] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13325.283180] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13325.286600] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13325.794068] LustreError: 411160:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13328.343975] Lustre: server umount lustre-OST0000 complete [13335.995846] Lustre: server umount lustre-MDT0000 complete [13340.340359] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [13341.347812] Key type lgssc unregistered [13341.477520] LNet: 413512:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13341.479944] LNetError: 413512:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13341.487431] LNet: Removed LNI 192.168.201.105@tcp [13341.774593] Key type .llcrypt unregistered [13341.775614] Key type ._llcrypt unregistered [13354.301735] Key type ._llcrypt registered [13354.302740] Key type .llcrypt registered [13354.341753] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [13354.447305] Key type .llcrypt unregistered [13354.448478] Key type ._llcrypt unregistered [13360.969319] Key type ._llcrypt registered [13360.970762] Key type .llcrypt registered [13361.008193] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [13361.341224] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13361.396260] alg: No test for adler32 (adler32-zlib) [13362.243156] Lustre: Lustre: Build Version: 2.17.51_1_gce01c51 [13362.324021] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [13363.904116] Key type lgssc registered [13364.302408] Lustre: Echo OBD driver; http://www.lustre.org/ [13373.039752] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13373.046141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13374.141124] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13374.152922] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13374.191085] Lustre: lustre-MDT0000: new disk, initializing [13374.215635] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13374.221923] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13375.547291] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [13379.708018] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13379.758255] Lustre: 416825:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13379.787985] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13379.789963] Lustre: Skipped 1 previous similar message [13379.824479] Lustre: lustre-MDT0001: new disk, initializing [13379.856699] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13379.868536] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13379.872175] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13381.142880] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [13384.252433] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13385.494699] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13387.764205] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13387.878211] Lustre: lustre-OST0000: new disk, initializing [13387.880378] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13387.914410] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13389.758171] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [13390.648622] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13390.652593] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13390.684650] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13392.829905] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13395.424907] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13395.427726] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13395.432228] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13395.938123] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13395.939516] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13395.941460] Lustre: Skipped 1 previous similar message [13401.058512] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13401.060644] Lustre: Skipped 3 previous similar messages [13401.302280] Lustre: server umount lustre-MDT0000 complete [13402.546285] LustreError: 417603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773620542 with bad export cookie 8041927549329027429 [13402.549698] LustreError: 417603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13402.549878] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13402.660022] Lustre: server umount lustre-MDT0001 complete [13413.088469] Lustre: server umount lustre-OST0000 complete [13416.619278] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 20:22:35 (1773620555) [13419.258496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13419.437979] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13420.716417] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [13423.614164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13423.732367] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13425.025016] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [13426.854747] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13428.132702] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13443.556225] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13443.626880] Lustre: Found index 0 for lustre-OST0000, updating log [13443.691752] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13445.568353] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [13445.738427] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13447.290198] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13449.506842] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13449.508179] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13449.513979] Lustre: Skipped 1 previous similar message [13454.305797] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13454.308036] Lustre: Skipped 3 previous similar messages [13454.605546] Lustre: server umount lustre-OST0000 complete [13470.176926] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13470.179513] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13470.184651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13473.310081] Lustre: server umount lustre-MDT0000 complete [13474.541561] LustreError: 421196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773620614 with bad export cookie 8041927549329028808 [13474.545365] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13474.545573] LustreError: 421196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13474.672569] Lustre: server umount lustre-MDT0001 complete [13477.445474] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 20:23:36 (1773620616) [13511.840326] LNet: 1 local NIs in recovery (showing 1): 192.168.201.105@tcp