[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 661547621 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2576MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.002262] x2apic enabled [ 0.003004] Switched APIC routing to physical x2apic. [ 0.004008] kvm-guest: setup PV IPIs [ 0.006681] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007017] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008007] pid_max: default: 32768 minimum: 301 [ 0.009132] LSM: Security Framework initializing [ 0.010037] Yama: becoming mindful. [ 0.011022] SELinux: Initializing. [ 0.012053] *** VALIDATE selinux *** [ 0.019000] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.023263] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.024138] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.025088] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026088] *** VALIDATE tmpfs *** [ 0.028336] *** VALIDATE proc *** [ 0.029171] *** VALIDATE cgroup *** [ 0.030006] *** VALIDATE cgroup2 *** [ 0.031241] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.032131] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.033004] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.034023] Spectre V2 : User space: Vulnerable [ 0.035005] Speculative Store Bypass: Vulnerable [ 0.037773] debug: unmapping init [mem 0xffffffff97859000-0xffffffff97860fff] [ 0.039279] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.040753] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.041014] ... version: 2 [ 0.041980] ... bit width: 48 [ 0.042009] ... generic registers: 4 [ 0.043008] ... value mask: 0000ffffffffffff [ 0.044007] ... max period: 00007fffffffffff [ 0.045008] ... fixed-purpose events: 3 [ 0.046018] ... event mask: 000000070000000f [ 0.047366] rcu: Hierarchical SRCU implementation. [ 0.049452] smp: Bringing up secondary CPUs ... [ 0.050581] x86: Booting SMP configuration: [ 0.051020] .... node #0, CPUs: #1 #2 #3 [ 0.054710] smp: Brought up 1 node, 4 CPUs [ 0.056009] smpboot: Max logical packages: 1 [ 0.057014] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.106296] node 0 deferred pages initialised in 47ms [ 0.114131] devtmpfs: initialized [ 0.115331] x86/mm: Memory block size: 128MB [ 0.119586] gcov: version magic: 0x41383552 [ 0.122319] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.123086] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.125251] pinctrl core: initialized pinctrl subsystem [ 0.128101] [ 0.129010] ************************************************************* [ 0.134029] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.138010] ** ** [ 0.142021] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.145011] ** ** [ 0.148013] ** This means that this kernel is built to expose internal ** [ 0.152013] ** IOMMU data structures, which may compromise security on ** [ 0.156013] ** your system. ** [ 0.163019] ** ** [ 0.172033] ** If you see this message and you are not debugging the ** [ 0.176014] ** kernel, report this immediately to your vendor! ** [ 0.178012] ** ** [ 0.181011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.185014] ************************************************************* [ 0.189343] NET: Registered protocol family 16 [ 0.193146] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.197093] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.200233] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.211037] cpuidle: using governor menu [ 0.212776] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.215778] PCI: Using configuration type 1 for base access [ 0.218106] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.227151] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.228017] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.230140] cryptd: max_cpu_qlen set to 1000 [ 0.232598] ACPI: Added _OSI(Module Device) [ 0.233050] ACPI: Added _OSI(Processor Device) [ 0.235011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.236009] ACPI: Added _OSI(Processor Aggregator Device) [ 0.240000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.246006] ACPI: Interpreter enabled [ 0.248082] ACPI: PM: (supports S0 S3 S4 S5) [ 0.249010] ACPI: Using IOAPIC for interrupt routing [ 0.251183] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.254418] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.265276] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.267025] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.270018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.274080] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.279751] acpiphp: Slot [2] registered [ 0.281092] acpiphp: Slot [5] registered [ 0.283188] acpiphp: Slot [6] registered [ 0.285092] acpiphp: Slot [7] registered [ 0.286088] acpiphp: Slot [8] registered [ 0.288088] acpiphp: Slot [9] registered [ 0.289095] acpiphp: Slot [10] registered [ 0.291081] acpiphp: Slot [3] registered [ 0.292057] acpiphp: Slot [4] registered [ 0.293059] acpiphp: Slot [11] registered [ 0.294061] acpiphp: Slot [12] registered [ 0.295074] acpiphp: Slot [13] registered [ 0.296251] acpiphp: Slot [14] registered [ 0.297177] acpiphp: Slot [15] registered [ 0.298195] acpiphp: Slot [16] registered [ 0.299156] acpiphp: Slot [17] registered [ 0.301062] acpiphp: Slot [18] registered [ 0.302068] acpiphp: Slot [19] registered [ 0.303068] acpiphp: Slot [20] registered [ 0.304089] acpiphp: Slot [21] registered [ 0.305048] acpiphp: Slot [22] registered [ 0.306058] acpiphp: Slot [23] registered [ 0.307055] acpiphp: Slot [24] registered [ 0.308053] acpiphp: Slot [25] registered [ 0.309125] acpiphp: Slot [26] registered [ 0.310119] acpiphp: Slot [27] registered [ 0.311069] acpiphp: Slot [28] registered [ 0.312065] acpiphp: Slot [29] registered [ 0.313084] acpiphp: Slot [30] registered [ 0.314245] acpiphp: Slot [31] registered [ 0.316064] PCI host bridge to bus 0000:00 [ 0.319022] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.322018] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.325024] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.327023] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.329023] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.331028] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.333147] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.336129] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.339429] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.350015] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.353571] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.355010] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.356014] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.359017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.362575] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.364556] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.367061] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.370653] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.374014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.383886] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.386010] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.388671] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.394013] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.400013] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.429019] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.441633] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.448027] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.457069] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.474012] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.482897] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.488012] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.493013] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.504000] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.504000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.504000] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.504000] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.508018] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.513000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.515000] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.516000] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.533017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.542953] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.554018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.562017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.580014] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.591742] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.595587] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.599403] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.604141] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.605183] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.612029] iommu: Default domain type: Passthrough [ 0.614521] SCSI subsystem initialized [ 0.615107] ACPI: bus type USB registered [ 0.616085] usbcore: registered new interface driver usbfs [ 0.618054] usbcore: registered new interface driver hub [ 0.620071] usbcore: registered new device driver usb [ 0.621000] pps_core: LinuxPPS API ver. 1 registered [ 0.621000] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.623042] PTP clock support registered [ 0.625039] EDAC MC: Ver: 3.0.0 [ 0.626107] PCI: Using ACPI for IRQ routing [ 0.627639] NetLabel: Initializing [ 0.628010] NetLabel: domain hash size = 128 [ 0.629008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.631082] NetLabel: unlabeled traffic allowed by default [ 0.633076] vgaarb: loaded [ 0.635454] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.636045] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.644494] clocksource: Switched to clocksource kvm-clock [ 0.767238] VFS: Disk quotas dquot_6.6.0 [ 0.768926] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.772782] *** VALIDATE ramfs *** [ 0.774199] *** VALIDATE hugetlbfs *** [ 0.775860] pnp: PnP ACPI init [ 0.778229] pnp: PnP ACPI: found 6 devices [ 0.802514] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.806062] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.808213] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.810307] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.813398] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.815896] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.819242] NET: Registered protocol family 2 [ 0.821933] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.828304] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.833015] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.841179] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.846520] TCP: Hash tables configured (established 65536 bind 65536) [ 0.851043] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.857829] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.862415] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.867856] NET: Registered protocol family 1 [ 0.873067] RPC: Registered named UNIX socket transport module. [ 0.876742] RPC: Registered udp transport module. [ 0.878823] RPC: Registered tcp transport module. [ 0.880851] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.883507] NET: Registered protocol family 44 [ 0.885280] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.888406] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.890733] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.893084] PCI: CLS 0 bytes, default 64 [ 0.895456] Unpacking initramfs... [ 2.982362] debug: unmapping init [mem 0xffff8cf67cc54000-0xffff8cf67ffbffff] [ 2.986263] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.988300] software IO TLB: mapped [mem 0x00000000b8c54000-0x00000000bcc54000] (64MB) [ 2.990528] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.824225] Initialise system trusted keyrings [ 3.825563] Key type blacklist registered [ 3.826995] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.835341] zbud: loaded [ 3.837867] *** VALIDATE nfs *** [ 3.838803] *** VALIDATE nfs4 *** [ 3.839956] pstore: using deflate compression [ 3.843124] Platform Keyring initialized [ 3.953432] NET: Registered protocol family 38 [ 3.955932] Key type asymmetric registered [ 3.959852] Asymmetric key parser 'x509' registered [ 3.962538] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.971320] io scheduler mq-deadline registered [ 3.973160] io scheduler kyber registered [ 3.975275] io scheduler bfq registered [ 3.977922] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.981584] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.985272] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.988326] ACPI: Power Button [PWRF] [ 3.994337] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.005314] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.047164] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.054874] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.082077] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.111662] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.140631] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.145221] Non-volatile memory driver v1.3 [ 4.146655] Linux agpgart interface v0.103 [ 4.180835] virtio_blk virtio1: [vda] 134712 512-byte logical blocks (69.0 MB/65.8 MiB) [ 4.182683] vda: detected capacity change from 0 to 68972544 [ 4.206078] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.208507] vdb: detected capacity change from 0 to 1073741824 [ 4.239941] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.244059] vdc: detected capacity change from 0 to 2621440000 [ 4.273052] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.275889] vdd: detected capacity change from 0 to 2621440000 [ 4.306768] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.309419] vde: detected capacity change from 0 to 4294967296 [ 4.341831] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.344382] vdf: detected capacity change from 0 to 4294967296 [ 4.360873] libphy: Fixed MDIO Bus: probed [ 4.370779] usbcore: registered new interface driver usbserial_generic [ 4.373316] usbserial: USB Serial support registered for generic [ 4.375673] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.380216] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.381724] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.383786] mousedev: PS/2 mouse device common for all mice [ 4.388191] rtc_cmos 00:05: RTC can wake from S4 [ 4.390909] rtc_cmos 00:05: registered as rtc0 [ 4.392405] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.394499] intel_pstate: CPU model not supported [ 4.397290] hid: raw HID events driver (C) Jiri Kosina [ 4.398802] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.399921] usbcore: registered new interface driver usbhid [ 4.399925] usbhid: USB HID core driver [ 4.400681] drop_monitor: Initializing network drop monitor service [ 4.413618] Initializing XFRM netlink socket [ 4.415388] NET: Registered protocol family 10 [ 4.418810] Segment Routing with IPv6 [ 4.420354] NET: Registered protocol family 17 [ 4.422574] mpls_gso: MPLS GSO support [ 4.422988] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.428277] RAS: Correctable Errors collector initialized. [ 4.428956] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.430569] AVX version of gcm_enc/dec engaged. [ 4.430573] AES CTR mode by8 optimization enabled [ 4.535637] sched_clock: Marking stable (4535488868, 0)->(5695731352, -1160242484) [ 4.539225] registered taskstats version 1 [ 4.540897] Loading compiled-in X.509 certificates [ 4.542896] zswap: loaded using pool lzo/zbud [ 4.579432] Key type big_key registered [ 4.624472] Key type encrypted registered [ 4.628154] ima: No TPM chip found, activating TPM-bypass! [ 4.633519] ima: Allocated hash algorithm: sha1 [ 4.637308] ima: No architecture policies found [ 4.639927] evm: Initialising EVM extended attributes: [ 4.645142] evm: security.selinux [ 4.647902] evm: security.ima [ 4.651395] evm: security.capability [ 4.654871] evm: HMAC attrs: 0x1 [ 4.661366] rtc_cmos 00:05: setting system clock to 2026-03-05 02:37:22 UTC (1772678242) [ 4.678293] debug: unmapping init [mem 0xffffffff98803000-0xffffffff989fffff] [ 4.683685] debug: unmapping init [mem 0xffffffff97582000-0xffffffff97858fff] [ 4.691218] Write protecting the kernel read-only data: 28672k [ 4.695844] debug: unmapping init [mem 0xffffffff95c03000-0xffffffff95dfffff] [ 4.700064] debug: unmapping init [mem 0xffffffff96514000-0xffffffff965fffff] [ 4.759206] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.773476] systemd[1]: Detected virtualization kvm. [ 4.776823] systemd[1]: Detected architecture x86-64. [ 4.780831] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.816683] systemd[1]: No hostname configured. [ 4.818639] systemd[1]: Set hostname to . [ 4.821674] random: systemd: uninitialized urandom read (16 bytes read) [ 4.824847] systemd[1]: Initializing machine ID from random generator. [ 4.944390] random: ln: uninitialized urandom read (6 bytes read) [ 5.212472] random: systemd: uninitialized urandom read (16 bytes read) [ 5.215339] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 5.227810] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 5.233433] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket. Starting Journal Service... Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. Starting Setup Virtual Console... [ OK ] Reached target Timers. Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Swap. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.546776] device-mapper: uevent: version 1.0.3 [ 6.548402] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 7.681194] virtio_net virtio0 ens2: renamed from eth0 [ 7.701450] random: fast init done [ 8.036724] scsi host0: ata_piix [ 8.095959] scsi host1: ata_piix [ 8.102587] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 8.112691] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 13.170240] random: crng init done [ 13.172302] random: 7 urandom warning(s) missed due to ratelimiting [ 15.452725] dracut-initqueue[587]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 17.307979] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 19.932807] printk: systemd: 25 output lines suppressed due to ratelimiting [ 20.784638] SELinux: Disabled at runtime. [ 20.870729] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 20.880469] systemd[1]: Detected virtualization kvm. [ 20.882859] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 22.361627] systemd[1]: initrd-switch-root.service: Succeeded. [ 22.366420] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 22.373713] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 22.376754] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 22.379625] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 22.389082] systemd[1]: Starting Journal Service... Starting Journal Service... [ 22.395261] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Switch Root. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Initrd File Systems. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-getty.slice. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Initrd Root File System. Mounting POSIX Message Queue File System... Mounting Huge Pages File System... [ OK ] Reached target Local Encrypted Volumes. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ 22.772891] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Kernel Debug File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Apply Kernel Variables... [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 23.877368] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 25.218994] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 25.251483] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 27.208232] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 27.525471] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ ***] A start job is running for Configur…only root support (10s / no limit) [ **] A start job is running for Configur…only root support (11s / no limit) [ *] A start job is running for Configur…only root support (11s / no limit) [ **] A start job is running for Configur…only root support (12s / no limit)[ 34.684482] Key type dns_resolver registered [[ 35.157115] NFS: Registering the id_resolver key type [ 35.158880] Key type id_resolver registered [ 35.160327] Key type id_legacy registered ***] A start job is running for Configur…only root support (12s / no limit) [ *** ] A start job is running for Configur…only root support (13s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Mark the need to relabel after reboot. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Authorization Manager. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg103-server login: [ 65.656032] libcfs: loading out-of-tree module taints kernel. [ 65.673897] Key type ._llcrypt registered [ 65.675095] Key type .llcrypt registered [ 65.720374] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 73.989187] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 74.612915] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 74.627537] alg: No test for adler32 (adler32-zlib) [ 75.616243] Lustre: Lustre: Build Version: 2.17.50_193_g407ab0b [ 75.908694] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 77.520180] Key type lgssc registered [ 78.197751] Lustre: Echo OBD driver; http://www.lustre.org/ [ 86.649883] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 106.048153] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 111.917589] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 111.930203] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 113.048593] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 113.065576] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 113.110196] Lustre: lustre-MDT0000: new disk, initializing [ 113.157569] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 113.168481] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 114.941680] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 121.159845] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 121.223387] Lustre: 6474:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 121.237962] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 121.240387] Lustre: Skipped 1 previous similar message [ 121.289568] Lustre: lustre-MDT0001: new disk, initializing [ 121.321679] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 121.339404] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 121.343208] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 123.085892] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 125.716784] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 129.363377] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 129.463678] Lustre: lustre-OST0000: new disk, initializing [ 129.465688] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 129.497561] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 131.658989] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 132.847237] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 132.850571] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 132.877785] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 137.069983] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 137.115672] Lustre: lustre-OST0001: new disk, initializing [ 137.117494] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 137.133877] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 139.075906] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 140.395070] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 140.397631] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 140.410452] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 145.526518] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 150.906168] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 158.805948] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing check_logdir /tmp/testlogs/ [ 161.324876] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing yml_node [ 163.653072] Lustre: DEBUG MARKER: Client: 2.17.50.193 [ 165.054819] Lustre: DEBUG MARKER: MDS: 2.17.50.193 [ 166.345533] Lustre: DEBUG MARKER: OSS: 2.17.50.193 [ 167.155205] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Mar 4 21:40:05 EST 2026 [ 176.054098] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 177.066103] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 191.976909] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 191.979135] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 191.986702] Lustre: Skipped 3 previous similar messages [ 192.000903] Lustre: Skipped 3 previous similar messages [ 196.678100] Lustre: server umount lustre-MDT0000 complete [ 199.786885] LustreError: 6465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772678437 with bad export cookie 5421713292659571204 [ 199.787132] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 199.794891] LustreError: 6465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 202.209906] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 202.210881] LustreError: 6482:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 202.220755] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 202.235625] LustreError: 6482:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 202.238437] Lustre: Skipped 1 previous similar message [ 204.257118] LustreError: 6480:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 204.257369] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 204.269819] LustreError: 6480:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 204.277305] Lustre: Skipped 1 previous similar message [ 206.112874] Lustre: server umount lustre-MDT0001 complete [ 208.883894] Lustre: server umount lustre-OST0000 complete [ 211.467691] Lustre: server umount lustre-OST0001 complete [ 215.650383] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 222.093197] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 246.581947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 246.806926] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 246.837852] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 246.898438] Lustre: lustre-MDT0000: new disk, initializing [ 246.967424] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 246.989666] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 249.742952] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 257.047032] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 257.125223] Lustre: 14943:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 257.161157] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 257.163036] Lustre: Skipped 1 previous similar message [ 257.215124] Lustre: lustre-MDT0001: new disk, initializing [ 257.276952] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 257.304953] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 257.316978] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 260.138705] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 265.303936] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 268.077856] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 272.343282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 274.482894] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 274.488354] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 274.577233] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 276.133025] hrtimer: interrupt took 7017669 ns [ 276.894454] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 282.862840] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 286.503244] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 286.697246] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 289.756878] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 289.881435] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 294.882967] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 294.887828] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 294.896916] Lustre: Skipped 1 previous similar message [ 294.900683] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 295.400876] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 297.380641] Lustre: server umount lustre-OST0000 complete [ 300.516708] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 300.522363] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 300.527753] Lustre: Skipped 2 previous similar messages [ 300.533783] Lustre: Skipped 2 previous similar messages [ 306.442646] Lustre: server umount lustre-MDT0000 complete [ 309.646558] LustreError: 14934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772678547 with bad export cookie 5421713292659572744 [ 309.653438] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 309.659189] LustreError: 14934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 309.942500] Lustre: server umount lustre-MDT0001 complete [ 313.258409] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 21:42:31 (1772678551) [ 314.907484] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 316.595377] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 21:42:34 (1772678554) [ 322.110759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 322.513987] LustreError: 18944:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 322.602878] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 322.615814] Lustre: Skipped 1 previous similar message [ 326.377967] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 327.650622] LustreError: 18945:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 332.691590] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 332.770167] LustreError: 18944:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 335.684070] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 339.405088] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 341.736741] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 346.313649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 350.476306] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 353.865901] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 362.980464] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 362.984968] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 362.994038] LustreError: Skipped 1 previous similar message [ 363.000597] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 363.009503] Lustre: Skipped 1 previous similar message [ 363.015359] Lustre: Skipped 2 previous similar messages [ 367.948739] Lustre: server umount lustre-OST0000 complete [ 373.227726] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 373.247428] Lustre: Skipped 1 previous similar message [ 376.947550] Lustre: server umount lustre-MDT0000 complete [ 378.337776] LustreError: 18944:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 378.352967] LustreError: 18944:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 379.748758] LustreError: 18926:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772678617 with bad export cookie 5421713292659573626 [ 379.754502] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 379.760072] LustreError: 18926:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 379.979655] Lustre: server umount lustre-MDT0001 complete [ 386.389549] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 388.312291] Key type lgssc unregistered [ 388.561899] LNet: 22123:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 388.570816] LNetError: 22123:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 388.596115] LNet: Removed LNI 192.168.201.103@tcp [ 389.319158] Key type .llcrypt unregistered [ 389.324063] Key type ._llcrypt unregistered [ 405.159266] Key type ._llcrypt registered [ 405.161375] Key type .llcrypt registered [ 405.250472] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 406.018319] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 406.027593] alg: No test for adler32 (adler32-zlib) [ 406.977091] Lustre: Lustre: Build Version: 2.17.50_193_g407ab0b [ 407.142611] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 408.824225] Key type lgssc registered [ 409.680746] Lustre: Echo OBD driver; http://www.lustre.org/ [ 415.263075] loop: module loaded [ 415.308445] loop0: detected capacity change from 0 to 204800000 [ 415.466163] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 415.478865] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 415.660418] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 419.411778] loop0: detected capacity change from 0 to 204800000 [ 419.536236] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 419.542880] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 419.550422] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 423.045624] loop0: detected capacity change from 0 to 204800000 [ 426.784291] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 426.808350] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 428.044070] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 428.101611] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 428.153896] Lustre: lustre-MDT0000: new disk, initializing [ 428.233380] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 428.280037] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 431.604798] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 438.316350] loop1: detected capacity change from 0 to 204800000 [ 442.168961] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 442.329276] Lustre: lustre-OST07c6: new disk, initializing [ 442.333781] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 442.338839] Lustre: Skipped 1 previous similar message [ 442.345553] Lustre: lustre-OST07c6: Not available for connect from 0@lo (not set up) [ 442.376724] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 446.004377] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 447.524190] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 447.531708] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 447.558529] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 450.348991] Lustre: 24255:0:(osd_internal.h:1455:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 450.367197] Lustre: 24255:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 450.378932] Lustre: 24255:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 450.391453] Lustre: 24255:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 450.398603] Lustre: 24255:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 450.405442] Lustre: 24255:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 469.655713] Lustre: 25709:0:(osd_internal.h:1455:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 469.658619] Lustre: 25709:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 469.661092] Lustre: 25709:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 469.665921] Lustre: 25709:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/4 [ 469.674493] Lustre: 25709:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 469.683191] Lustre: 25709:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 473.606682] Lustre: 24256:0:(osd_internal.h:1455:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 473.612950] Lustre: 24256:0:(osd_internal.h:1455:osd_trans_exec_op()) Skipped 1 previous similar message [ 473.618589] Lustre: 24256:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 473.623062] Lustre: 24256:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 473.629635] Lustre: 24256:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 473.637572] Lustre: 24256:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 473.641993] Lustre: 24256:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/0, punch: 0/0/0, quota 4/150/0 [ 473.648275] Lustre: 24256:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 473.652291] Lustre: 24256:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 473.657198] Lustre: 24256:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 473.660920] Lustre: 24256:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 473.666721] Lustre: 24256:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 476.823637] Lustre: 24255:0:(osd_internal.h:1455:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 476.829562] Lustre: 24255:0:(osd_internal.h:1455:osd_trans_exec_op()) Skipped 1 previous similar message [ 476.833408] Lustre: 24255:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 476.838286] Lustre: 24255:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 476.842288] Lustre: 24255:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 476.846402] Lustre: 24255:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 476.849378] Lustre: 24255:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/2 [ 476.852907] Lustre: 24255:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 476.856627] Lustre: 24255:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 476.860677] Lustre: 24255:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 476.864179] Lustre: 24255:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 476.866560] Lustre: 24255:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 480.994032] Lustre: 24255:0:(osd_internal.h:1455:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 480.998137] Lustre: 24255:0:(osd_internal.h:1455:osd_trans_exec_op()) Skipped 1 previous similar message [ 481.001958] Lustre: 24255:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 481.004893] Lustre: 24255:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 481.008145] Lustre: 24255:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 481.011191] Lustre: 24255:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 481.015490] Lustre: 24255:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/2 [ 481.019094] Lustre: 24255:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 481.024028] Lustre: 24255:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 481.027521] Lustre: 24255:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 481.031791] Lustre: 24255:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 481.034711] Lustre: 24255:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 490.354731] Lustre: 24257:0:(osd_internal.h:1455:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 490.357199] Lustre: 24257:0:(osd_internal.h:1455:osd_trans_exec_op()) Skipped 5 previous similar messages [ 490.359872] Lustre: 24257:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 490.362937] Lustre: 24257:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 490.365267] Lustre: 24257:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 490.368365] Lustre: 24257:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 490.372090] Lustre: 24257:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/0, punch: 0/0/0, quota 4/150/0 [ 490.375578] Lustre: 24257:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 490.378565] Lustre: 24257:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 490.382048] Lustre: 24257:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 490.386881] Lustre: 24257:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 490.389773] Lustre: 24257:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 538.984958] Lustre: server umount lustre-OST07c6 complete [ 554.074536] Lustre: server umount lustre-MDT0000 complete [ 564.286707] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 565.789713] Key type lgssc unregistered [ 565.939164] LNet: 27795:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 565.944511] LNetError: 27795:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 565.951655] LNet: Removed LNI 192.168.201.103@tcp [ 566.365934] Key type .llcrypt unregistered [ 566.368014] Key type ._llcrypt unregistered [ 576.764954] Key type ._llcrypt registered [ 576.766418] Key type .llcrypt registered [ 576.826434] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 577.444152] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 577.459900] alg: No test for adler32 (adler32-zlib) [ 578.376341] Lustre: Lustre: Build Version: 2.17.50_193_g407ab0b [ 578.489968] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 580.088130] Key type lgssc registered [ 580.663935] Lustre: Echo OBD driver; http://www.lustre.org/ [ 584.636429] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 584.646803] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 585.918206] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 587.944208] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 592.894269] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 593.173494] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 595.155864] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 598.429240] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 600.356601] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 604.472648] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 604.690152] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 608.085356] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 610.859206] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 611.203776] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 625.123355] LustreError: lustre-OST0000-osc-MDT0000: operation ost_setattr to node 0@lo failed: rc = -107 [ 625.123584] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 625.125733] LustreError: Skipped 1 previous similar message [ 625.132032] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 625.892341] Lustre: server umount lustre-OST0000 complete [ 628.196598] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 628.202218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 628.207181] Lustre: Skipped 1 previous similar message [ 628.215568] Lustre: Skipped 1 previous similar message [ 633.315905] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 633.319509] Lustre: Skipped 1 previous similar message [ 633.914124] Lustre: server umount lustre-MDT0000 complete [ 635.926936] LustreError: 29007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772678873 with bad export cookie 5376229134178357394 [ 635.929693] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 635.933892] LustreError: 29007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 646.203044] Lustre: server umount lustre-MDT0001 complete [ 652.337588] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 654.027432] Key type lgssc unregistered [ 654.225249] LNet: 32406:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 654.235487] LNetError: 32406:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 654.252841] LNet: Removed LNI 192.168.201.103@tcp [ 654.773488] Key type .llcrypt unregistered [ 654.778261] Key type ._llcrypt unregistered [ 667.024220] Key type ._llcrypt registered [ 667.028757] Key type .llcrypt registered [ 667.130265] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 667.787523] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 667.812934] alg: No test for adler32 (adler32-zlib) [ 668.754174] Lustre: Lustre: Build Version: 2.17.50_193_g407ab0b [ 668.911811] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 670.528563] Key type lgssc registered [ 671.240444] Lustre: Echo OBD driver; http://www.lustre.org/ [ 675.463961] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 675.475380] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 676.718855] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 678.899600] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 683.255882] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 683.452531] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 685.550587] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 688.664489] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 690.653900] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 694.185955] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 694.366448] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 697.261702] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 700.001275] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 701.562097] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 717.281173] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 717.284198] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 717.291185] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 719.335123] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 719.335300] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 719.335360] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 719.338702] Lustre: Skipped 1 previous similar message [ 722.277938] Lustre: server umount lustre-OST0000 complete [ 724.451377] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 724.455495] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 724.457220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 724.466027] Lustre: Skipped 1 previous similar message [ 729.569983] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 729.574032] Lustre: Skipped 2 previous similar messages [ 730.556600] Lustre: server umount lustre-MDT0000 complete [ 732.618789] LustreError: 33619:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772678970 with bad export cookie 8118627851717485010 [ 732.622787] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 732.629204] LustreError: 33619:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 732.821326] Lustre: server umount lustre-MDT0001 complete [ 738.328698] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 739.839034] Key type lgssc unregistered [ 740.029452] LNet: 36839:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 740.032890] LNetError: 36839:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 740.041338] LNet: Removed LNI 192.168.201.103@tcp [ 740.477531] Key type .llcrypt unregistered [ 740.479202] Key type ._llcrypt unregistered [ 753.248431] Key type ._llcrypt registered [ 753.250433] Key type .llcrypt registered [ 753.316139] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 753.868910] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 753.916416] alg: No test for adler32 (adler32-zlib) [ 754.849718] Lustre: Lustre: Build Version: 2.17.50_193_g407ab0b [ 755.033861] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 756.640129] Key type lgssc registered [ 757.201407] Lustre: Echo OBD driver; http://www.lustre.org/ [ 761.117288] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 761.131282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 762.394302] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 764.632111] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 769.292302] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 769.539179] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 771.691034] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 774.489474] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 776.583628] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 780.779697] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 781.082470] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 784.944984] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 787.248814] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 788.165491] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 799.626587] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.3@tcp (stopping) [ 799.719987] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 799.731275] Lustre: Skipped 2 previous similar messages [ 801.951507] Lustre: server umount lustre-MDT0000 complete [ 804.242209] LustreError: 38197:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772679042 with bad export cookie 4132115217422643100 [ 804.243711] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 804.248615] LustreError: 38197:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 804.729059] LustreError: 38209:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 804.740589] LustreError: 38209:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 804.750138] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.3@tcp (stopping) [ 804.752957] Lustre: Skipped 3 previous similar messages [ 804.834890] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 809.843324] LustreError: 40265:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 809.843339] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.3@tcp (stopping) [ 809.843348] Lustre: Skipped 1 previous similar message [ 809.853144] LustreError: 40265:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 810.553747] Lustre: server umount lustre-MDT0001 complete [ 829.412319] Lustre: server umount lustre-OST0000 complete [ 837.631986] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 839.021181] Key type lgssc unregistered [ 839.170753] LNet: 41717:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 839.175207] LNetError: 41717:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 839.183534] LNet: Removed LNI 192.168.201.103@tcp [ 839.571572] Key type .llcrypt unregistered [ 839.573693] Key type ._llcrypt unregistered [ 850.741306] Key type ._llcrypt registered [ 850.743426] Key type .llcrypt registered [ 850.793101] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 851.239700] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 851.302129] alg: No test for adler32 (adler32-zlib) [ 852.185618] Lustre: Lustre: Build Version: 2.17.50_193_g407ab0b [ 852.298780] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 853.904119] Key type lgssc registered [ 854.347262] Lustre: Echo OBD driver; http://www.lustre.org/ [ 858.008167] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 858.017215] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 859.249605] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 861.336483] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 865.101301] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 865.279709] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 866.926017] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 869.241163] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 870.923811] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 874.030733] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 874.183683] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 876.621656] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 878.859902] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 884.458436] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 894.343368] Lustre: lustre-OST0000: Not available for connect from 192.168.201.3@tcp (stopping) [ 894.947219] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 894.948525] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 894.948711] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 894.951641] Lustre: Skipped 1 previous similar message [ 898.033287] Lustre: server umount lustre-OST0000 complete [ 911.686812] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 911.693426] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 911.695379] Lustre: Skipped 1 previous similar message [ 914.786099] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 914.786527] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 914.797536] Lustre: Skipped 1 previous similar message [ 918.001187] Lustre: server umount lustre-MDT0000 complete [ 919.696425] LustreError: 43075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772679157 with bad export cookie 8562948754980396465 [ 919.700857] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 919.701505] LustreError: 43075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 925.906180] Lustre: server umount lustre-MDT0001 complete [ 930.587933] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 931.902461] Key type lgssc unregistered [ 932.075473] LNet: 46472:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 932.077940] LNetError: 46472:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 932.091413] LNet: Removed LNI 192.168.201.103@tcp [ 932.493332] Key type .llcrypt unregistered [ 932.495446] Key type ._llcrypt unregistered [ 944.738604] Key type ._llcrypt registered [ 944.741146] Key type .llcrypt registered [ 944.794371] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 945.412248] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 945.422922] alg: No test for adler32 (adler32-zlib) [ 946.357164] Lustre: Lustre: Build Version: 2.17.50_193_g407ab0b [ 946.481967] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 948.096158] Key type lgssc registered [ 948.690748] Lustre: Echo OBD driver; http://www.lustre.org/ [ 952.146920] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 952.161400] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 953.401408] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 955.225947] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 958.970572] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 959.159273] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 960.877187] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 963.146647] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 964.699767] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 967.489209] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 967.645725] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 969.915289] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 971.926334] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 978.923642] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 984.864286] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 986.431814] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 997.498153] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-03-0421h53m55s [ 998.346806] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 999.394339] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 999.397654] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 999.400547] Lustre: Skipped 2 previous similar messages [ 999.403366] Lustre: Skipped 1 previous similar message [ 1000.418448] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1000.421601] Lustre: Skipped 3 previous similar messages [ 1002.878390] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.3@tcp (stopping) [ 1005.493067] Lustre: server umount lustre-MDT0000 complete [ 1006.561765] LustreError: 47847:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1006.567796] LustreError: 47847:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1007.076835] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1008.948517] LustreError: 49587:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1010.271321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1010.350080] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1010.496417] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1010.539721] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1011.497301] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1012.280943] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1013.908248] Lustre: DEBUG MARKER: Wait for df (36007) ... [ 1014.570348] Lustre: DEBUG MARKER: done [ 1016.804399] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1016.806698] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1016.810503] Lustre: Skipped 1 previous similar message [ 1016.813364] Lustre: Skipped 1 previous similar message [ 1021.922739] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1021.928226] Lustre: Skipped 2 previous similar messages [ 1030.112198] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1030.176627] Lustre: server umount lustre-OST0000 complete [ 1032.161826] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1032.162710] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1032.169667] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1032.177106] Lustre: Skipped 3 previous similar messages [ 1037.858941] Lustre: server umount lustre-MDT0000 complete [ 1038.241467] LustreError: 47848:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1038.248788] LustreError: 47848:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1039.381371] LustreError: 49461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772679277 with bad export cookie 11819118580230690917 [ 1039.385583] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1039.394568] LustreError: 49461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1039.528048] Lustre: server umount lustre-MDT0001 complete [ 1043.399568] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 1044.407463] Key type lgssc unregistered [ 1044.521356] LNet: 52419:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1044.524974] LNetError: 52419:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1044.534243] LNet: Removed LNI 192.168.201.103@tcp [ 1044.821142] Key type .llcrypt unregistered [ 1044.822985] Key type ._llcrypt unregistered [ 1061.495532] Key type ._llcrypt registered [ 1061.497289] Key type .llcrypt registered [ 1061.551332] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 1062.131777] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1062.158529] alg: No test for adler32 (adler32-zlib) [ 1063.038586] Lustre: Lustre: Build Version: 2.17.50_193_g407ab0b [ 1063.148120] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 1064.736260] Key type lgssc registered [ 1065.135727] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1068.348514] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1068.356318] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1069.478488] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1069.488466] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1069.567266] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1071.325419] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1075.090779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1075.150793] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1075.166159] Lustre: 54584:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1075.318528] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1077.244262] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1079.775204] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1081.378216] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1084.398696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1084.500949] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1084.589500] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1087.121375] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1087.333345] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1089.374751] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1092.837448] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-03-0421h55m31s [ 1093.422550] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1096.261258] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1097.336088] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1097.354111] LustreError: 54593:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1117.568887] Lustre: lustre-MDT0000: Client c1fac284-7122-4a63-ab7d-f56ff3b5d20b (at 192.168.201.3@tcp) reconnecting [ 1137.400104] LustreError: 54593:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1138.283658] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1139.258511] Lustre: DEBUG MARKER: done [ 1142.754342] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 1142.758408] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1142.764236] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1143.779279] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1143.780483] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1143.780629] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1147.285548] Lustre: server umount lustre-OST0000 complete [ 1148.897333] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1148.897782] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1148.901876] Lustre: Skipped 1 previous similar message [ 1148.906264] Lustre: Skipped 2 previous similar messages [ 1154.779488] Lustre: server umount lustre-MDT0000 complete [ 1156.122237] LustreError: 54577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772679393 with bad export cookie 8927341518095078752 [ 1156.125890] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1156.127307] LustreError: 54577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1156.318831] Lustre: server umount lustre-MDT0001 complete [ 1160.491941] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 1161.790768] Key type lgssc unregistered [ 1161.946451] LNet: 58551:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1161.950317] LNetError: 58551:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1161.959419] LNet: Removed LNI 192.168.201.103@tcp [ 1162.344834] Key type .llcrypt unregistered [ 1162.346319] Key type ._llcrypt unregistered [ 1181.289532] Key type ._llcrypt registered [ 1181.290532] Key type .llcrypt registered [ 1181.338424] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 1181.753647] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1181.793049] alg: No test for adler32 (adler32-zlib) [ 1182.668266] Lustre: Lustre: Build Version: 2.17.50_193_g407ab0b [ 1182.761904] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 1184.352268] Key type lgssc registered [ 1184.797808] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1186.055726] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1186.066381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1187.180420] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1187.190808] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1187.262325] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1187.657582] Lustre: Failing over lustre-MDT0000 [ 1187.772482] Lustre: server umount lustre-MDT0000 complete [ 1190.612846] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 21:57:08 (1772679428) [ 1193.737724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1193.987430] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1195.713274] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1199.398295] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1199.448350] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1199.461891] Lustre: 61313:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1199.575615] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1201.169559] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1203.609317] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1205.192928] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1208.209859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1208.310459] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1208.394146] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1209.376088] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1210.911942] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1213.453601] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1216.752174] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1219.555766] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1219.558180] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1219.561789] Lustre: Skipped 2 previous similar messages [ 1219.564342] Lustre: Skipped 1 previous similar message [ 1224.673626] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1224.676176] Lustre: Skipped 3 previous similar messages [ 1225.001652] Lustre: server umount lustre-MDT0000 complete [ 1226.521208] LustreError: 62034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772679464 with bad export cookie 4112054969930784594 [ 1226.524342] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1226.526936] LustreError: 62034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1226.658202] Lustre: server umount lustre-MDT0001 complete [ 1228.122940] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1230.130517] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1233.621475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1236.960963] LustreError: 60234:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000089efc47d x1858789095538304/t0(0) o250->MGC192.168.201.103@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1237.091927] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1237.128513] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1237.165119] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1238.710501] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1242.503025] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1242.594085] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1242.598130] Lustre: Skipped 1 previous similar message [ 1244.228156] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1245.216533] Lustre: 60237:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772679467/real 1772679467] req@00000000699120ee x1858789095537920/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1772679483 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1246.392766] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1247.718902] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1247.893388] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1248.736361] Lustre: 60238:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772679469/real 1772679469] req@00000000fb323582 x1858789095538048/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1772679485 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1249.768480] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1251.104119] Lustre: 60235:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772679472/real 1772679472] req@0000000003d47ea0 x1858789095538176/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1772679488 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1252.833107] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1252.835789] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1252.841847] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1254.888534] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1254.896651] Lustre: Skipped 1 previous similar message [ 1256.239634] Lustre: server umount lustre-MDT0000 complete [ 1257.851265] LustreError: 61304:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772679495 with bad export cookie 4112054969930786869 [ 1257.854766] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1257.855251] LustreError: 61304:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1257.973233] Lustre: server umount lustre-MDT0001 complete [ 1260.684503] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1264.467515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1268.128865] LustreError: 60234:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000f3b4f213 x1858789095557888/t0(0) o250->MGC192.168.201.103@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1268.263376] LustreError: 62938:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1268.274291] LustreError: 62938:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1268.300577] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1268.303338] Lustre: Skipped 1 previous similar message [ 1268.332030] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1270.058859] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1273.316241] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1273.316265] LustreError: 65969:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1273.329619] LustreError: 65969:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1274.165852] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1275.360362] Lustre: 60237:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772679497/real 1772679497] req@000000004d4ffff0 x1858789095557504/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1772679513 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1275.368500] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1275.377143] Lustre: 60237:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1276.099202] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1278.664871] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1279.456289] Lustre: 60236:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772679501/real 1772679501] req@00000000d8e62ab7 x1858789095557760/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1772679517 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1279.464905] Lustre: 60236:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1280.220123] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1284.577190] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1284.580296] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1284.585484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1284.591897] Lustre: Skipped 3 previous similar messages [ 1288.808426] Lustre: server umount lustre-MDT0000 complete [ 1290.392265] LustreError: 62035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772679528 with bad export cookie 4112054969930788248 [ 1290.394354] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1290.400389] LustreError: 62035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1290.518548] Lustre: server umount lustre-MDT0001 complete [ 1293.111596] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1294.538829] Lustre: DEBUG MARKER: files compared the same [ 1305.634615] Lustre: server umount lustre-OST0000 complete [ 1311.812988] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 1313.006353] Key type lgssc unregistered [ 1313.138584] LNet: 70233:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1313.142224] LNetError: 70233:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1313.150313] LNet: Removed LNI 192.168.201.103@tcp [ 1313.459172] Key type .llcrypt unregistered [ 1313.461147] Key type ._llcrypt unregistered [ 1322.911498] Key type ._llcrypt registered [ 1322.912419] Key type .llcrypt registered [ 1322.956152] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 1323.334238] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1323.368177] alg: No test for adler32 (adler32-zlib) [ 1324.242472] Lustre: Lustre: Build Version: 2.17.50_193_g407ab0b [ 1324.341463] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 1325.928198] Key type lgssc registered [ 1326.408236] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1329.548068] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1329.556392] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1330.731465] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1332.125287] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug +malloc all [ 1335.239564] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1335.341763] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1336.772118] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug +malloc all [ 1338.835496] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1340.190965] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1342.518164] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1342.617971] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1344.578825] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug +malloc all [ 1346.584254] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1356.963031] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1361.377256] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1361.381451] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1361.387115] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1362.401193] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1362.401556] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1362.401721] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1365.832417] Lustre: server umount lustre-OST0000 complete [ 1367.520733] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1367.520946] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1367.524439] Lustre: Skipped 1 previous similar message [ 1367.527292] Lustre: Skipped 2 previous similar messages [ 1372.640435] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1372.642445] Lustre: Skipped 1 previous similar message [ 1373.264661] Lustre: server umount lustre-MDT0000 complete [ 1374.436719] LustreError: 71590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772679612 with bad export cookie 18152369916468461958 [ 1374.437838] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1374.439186] LustreError: 71590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1374.548672] Lustre: server umount lustre-MDT0001 complete [ 1377.879717] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 1378.863432] Key type lgssc unregistered [ 1378.987350] LNet: 74807:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1378.989809] LNetError: 74807:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1378.999309] LNet: Removed LNI 192.168.201.103@tcp [ 1379.250829] Key type .llcrypt unregistered [ 1379.251895] Key type ._llcrypt unregistered [ 1388.195815] Key type ._llcrypt registered [ 1388.196799] Key type .llcrypt registered [ 1388.237467] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 1388.721073] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1388.732226] alg: No test for adler32 (adler32-zlib) [ 1389.581295] Lustre: Lustre: Build Version: 2.17.50_193_g407ab0b [ 1389.674475] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 1391.256140] Key type lgssc registered [ 1391.627175] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1394.027911] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1394.033555] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1409.440328] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1414.560201] LustreError: 76157:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.103@tcp: failed processing log, type 4: rc = -110 [ 1441.184178] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1443.096611] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1444.886259] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1447.390938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1447.428707] Lustre: *** cfs_fail_loc=706, val=0*** [ 1447.567914] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1447.594247] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 1448.827415] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1451.638519] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1451.782436] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1453.142662] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1454.888753] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1456.100302] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1457.123446] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1457.129433] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1457.129494] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1458.896325] Lustre: server umount lustre-OST0000 complete [ 1462.242085] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1462.242375] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1462.243040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1462.243044] Lustre: Skipped 1 previous similar message [ 1462.247302] Lustre: Skipped 2 previous similar messages [ 1466.260702] Lustre: server umount lustre-MDT0000 complete [ 1467.361482] LustreError: 78694:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1467.369254] LustreError: 78694:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1467.565565] LustreError: 76163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772679705 with bad export cookie 7747462834779462525 [ 1467.568511] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1467.570720] LustreError: 76163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1467.691285] Lustre: server umount lustre-MDT0001 complete [ 1471.437292] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 1472.357388] Key type lgssc unregistered [ 1472.472336] LNet: 79428:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1472.474771] LNetError: 79428:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1472.486284] LNet: Removed LNI 192.168.201.103@tcp [ 1472.731030] Key type .llcrypt unregistered [ 1472.731844] Key type ._llcrypt unregistered [ 1481.595669] Key type ._llcrypt registered [ 1481.596855] Key type .llcrypt registered [ 1481.634402] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 1482.063842] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1482.071628] alg: No test for adler32 (adler32-zlib) [ 1482.921777] Lustre: Lustre: Build Version: 2.17.50_193_g407ab0b [ 1483.005108] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 1484.592122] Key type lgssc registered [ 1484.942715] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1487.507085] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1487.514557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1489.850508] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1492.398493] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1492.575154] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1493.845698] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1496.633954] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1496.750504] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1498.628778] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1501.514354] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1501.520544] Lustre: MGS: Client 996ef4eb-f50b-415e-b7c0-a82f2dbcfca8 (at 0@lo) reconnecting [ 1501.527812] LustreError: 81486:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 1501.609202] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1501.643972] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 1502.947300] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1511.905370] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1511.907636] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1511.911399] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1512.931938] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1512.932073] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1512.932401] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1512.934878] Lustre: Skipped 1 previous similar message [ 1516.821285] Lustre: server umount lustre-OST0000 complete [ 1518.051067] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1518.052723] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1518.052841] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1518.054542] Lustre: Skipped 1 previous similar message [ 1523.169141] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1523.171384] Lustre: Skipped 2 previous similar messages [ 1528.289332] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1528.291543] Lustre: Skipped 1 previous similar message [ 1532.384126] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1532.426957] Lustre: server umount lustre-MDT0000 complete [ 1533.743848] Lustre: server umount lustre-MDT0001 complete [ 1534.963834] Lustre: server umount MGS complete [ 1538.525545] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 22:02:56 (1772679776) [ 1547.751248] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 1550.739873] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 1562.246290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1563.513797] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1567.390405] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1567.475674] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 1567.495644] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1567.536820] Lustre: lustre-MDT0001: new disk, initializing [ 1567.562570] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1567.571149] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1567.572988] Lustre: 88532:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 1567.575055] LustreError: 88532:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 1568.875294] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1572.930887] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1573.028980] Lustre: lustre-OST0000: new disk, initializing [ 1573.031444] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1573.051376] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1574.957762] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1576.288797] Lustre: 89672:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@0000000021cf0138 x1858789410113152/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1772679830 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1577.992891] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1580.311208] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1580.316680] Lustre: MGS: Client fe0dce74-837e-439b-b875-69a29630be58 (at 0@lo) reconnecting [ 1580.336059] Lustre: 90213:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1580.345701] LustreError: 88518:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 1580.349888] LustreError: 88518:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 1580.356305] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1580.398777] Lustre: lustre-MDT0000: new disk, initializing [ 1580.465959] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1581.536633] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1581.537743] Lustre: 90228:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000660c95ba x1858789410118912/t0(0) o700->lustre-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1772679835 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1581.550080] Lustre: 90228:0:(client.c:1637:after_reply()) Skipped 1 previous similar message [ 1581.793748] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1583.587151] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1583.591061] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1583.593872] Lustre: Skipped 1 previous similar message [ 1583.597397] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 1600.994300] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1600.994869] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1600.996493] LustreError: Skipped 1 previous similar message [ 1601.002250] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1601.004422] Lustre: Skipped 3 previous similar messages [ 1606.289161] Lustre: server umount lustre-OST0000 complete [ 1609.185512] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1609.185821] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1609.191967] Lustre: Skipped 2 previous similar messages [ 1613.702092] Lustre: server umount lustre-MDT0000 complete [ 1614.306494] LustreError: 88503:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1614.311182] LustreError: 88503:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 1621.095809] Lustre: server umount lustre-MDT0001 complete [ 1622.335618] Lustre: server umount MGS complete [ 1626.014682] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 22:04:24 (1772679864) [ 1631.537534] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 1632.439559] Key type lgssc unregistered [ 1632.550270] LNet: 92704:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1632.552113] LNetError: 92704:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1632.560242] LNet: Removed LNI 192.168.201.103@tcp [ 1632.825105] Key type .llcrypt unregistered [ 1632.826568] Key type ._llcrypt unregistered [ 1639.140567] Key type ._llcrypt registered [ 1639.141639] Key type .llcrypt registered [ 1639.178384] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 1639.446815] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1639.453427] alg: No test for adler32 (adler32-zlib) [ 1640.292018] Lustre: Lustre: Build Version: 2.17.50_193_g407ab0b [ 1640.362669] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 1641.936128] Key type lgssc registered [ 1642.298891] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1644.899394] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1644.904887] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1646.062246] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1647.412499] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1650.339107] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1650.464501] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1651.739575] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1653.539280] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1654.728110] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1657.007619] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1657.123784] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1659.093286] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1660.827565] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1670.438997] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 1675.746493] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1675.748048] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1675.748529] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1675.750758] Lustre: Skipped 1 previous similar message [ 1675.753791] LustreError: Skipped 1 previous similar message [ 1675.918554] Lustre: server umount lustre-OST0000 complete [ 1679.200509] Lustre: server umount lustre-MDT0000 complete [ 1680.428854] LustreError: 93911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772679918 with bad export cookie 6600552058956267243 [ 1680.433504] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1680.547097] Lustre: server umount lustre-MDT0001 complete [ 1683.675712] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 1684.696841] Key type lgssc unregistered [ 1684.820589] LNet: 97083:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1684.822818] LNetError: 97083:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1684.829441] LNet: Removed LNI 192.168.201.103@tcp [ 1685.093606] Key type .llcrypt unregistered [ 1685.094964] Key type ._llcrypt unregistered [ 1691.023796] Key type ._llcrypt registered [ 1691.024692] Key type .llcrypt registered [ 1691.061402] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 1691.376240] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1691.440091] alg: No test for adler32 (adler32-zlib) [ 1692.285629] Lustre: Lustre: Build Version: 2.17.50_193_g407ab0b [ 1692.358579] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 1693.928175] Key type lgssc registered [ 1694.236226] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1696.089081] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1696.089258] LustreError: 98126:0:(tgt_mount.c:2535:server_fill_super()) cfs_race id 716 sleeping [ 1696.090786] Lustre: Skipped 1 previous similar message [ 1696.093448] LustreError: 98127:0:(tgt_mount.c:2535:server_fill_super()) cfs_fail_race id 716 waking [ 1696.096068] LustreError: 98126:0:(tgt_mount.c:2535:server_fill_super()) cfs_fail_race id 716 awake: rc=4996 [ 1696.097775] LustreError: 98126:0:(tgt_mount.c:2547:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 1696.101060] LustreError: 98126:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1696.101488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1697.256592] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1699.785133] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1699.888365] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1701.187440] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1702.742840] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing lsmod [ 1703.737456] LustreError: 99299:0:(tgt_mount.c:2535:server_fill_super()) cfs_race id 716 sleeping [ 1703.737528] LustreError: 99298:0:(tgt_mount.c:2535:server_fill_super()) cfs_fail_race id 716 waking [ 1703.744224] LustreError: 99299:0:(tgt_mount.c:2535:server_fill_super()) cfs_fail_race id 716 awake: rc=4997 [ 1703.748448] LustreError: 99299:0:(tgt_mount.c:2547:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 1703.751639] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1703.752484] LustreError: 99299:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1703.879580] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1704.796493] LustreError: 98194:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1704.799868] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1704.800896] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1704.801735] LustreError: 98194:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1704.804804] Lustre: Skipped 2 previous similar messages [ 1704.810250] LustreError: 98194:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 0, failed: rc = -5 [ 1709.922268] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1709.924507] Lustre: Skipped 4 previous similar messages [ 1715.169313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1715.171822] Lustre: Skipped 2 previous similar messages [ 1719.264117] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1719.319651] Lustre: server umount lustre-MDT0000 complete [ 1720.289826] LustreError: 98165:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1720.297603] LustreError: 98165:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1720.604297] LustreError: 98141:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772679958 with bad export cookie 10994534173222438991 [ 1720.608989] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1720.609240] LustreError: 98141:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1722.208192] LustreError: 98595:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000eca8606e x1858789630296192/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1722.246468] Lustre: server umount lustre-MDT0001 complete [ 1733.664483] Lustre: server umount lustre-OST0000 complete [ 1736.205721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1736.353415] LustreError: 100365:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1736.376945] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1737.567938] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1740.092962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1741.391130] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1743.012695] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1744.145789] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1746.180674] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1746.294113] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1746.296448] Lustre: Skipped 1 previous similar message [ 1747.302681] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 1748.056092] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1749.695987] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1752.544733] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1752.547125] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1752.551430] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1755.620063] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1755.623160] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1757.969101] Lustre: server umount lustre-OST0000 complete [ 1760.737735] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1760.738341] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1760.743823] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1760.745769] Lustre: Skipped 3 previous similar messages [ 1765.403140] Lustre: server umount lustre-MDT0000 complete [ 1765.857056] LustreError: 101076:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1765.865491] LustreError: 101076:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1766.670149] LustreError: 100345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772680004 with bad export cookie 10994534173222439649 [ 1766.673322] LustreError: 100345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1766.674123] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1766.792093] Lustre: server umount lustre-MDT0001 complete [ 1770.400043] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 1771.410308] Key type lgssc unregistered [ 1771.529272] LNet: 103542:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1771.531080] LNetError: 103542:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1771.536288] LNet: Removed LNI 192.168.201.103@tcp [ 1771.798522] Key type .llcrypt unregistered [ 1771.799477] Key type ._llcrypt unregistered [ 1780.385266] Key type ._llcrypt registered [ 1780.386360] Key type .llcrypt registered [ 1780.418870] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 1780.691020] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1780.726053] alg: No test for adler32 (adler32-zlib) [ 1781.563364] Lustre: Lustre: Build Version: 2.17.50_193_g407ab0b [ 1781.642949] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 1783.216162] Key type lgssc registered [ 1783.567824] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1785.932394] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1785.938179] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1787.092264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1788.423556] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1791.355298] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1791.490648] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1792.814564] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1794.629359] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1795.890595] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1798.327812] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1798.440275] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1800.329220] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1801.971538] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1812.773962] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 1819.291566] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 1819.945456] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 1822.177274] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1822.181508] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1822.188302] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1823.202635] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1823.202863] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1823.203094] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1823.205145] Lustre: Skipped 1 previous similar message [ 1826.837024] Lustre: server umount lustre-OST0000 complete [ 1828.321416] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1828.322055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1828.325509] Lustre: Skipped 1 previous similar message [ 1833.440976] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1833.442994] Lustre: Skipped 2 previous similar messages [ 1834.258210] Lustre: server umount lustre-MDT0000 complete [ 1835.371025] LustreError: 104899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772680073 with bad export cookie 8606708991131377204 [ 1835.373463] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1835.373858] LustreError: 104899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1835.471506] Lustre: server umount lustre-MDT0001 complete [ 1838.660083] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 1839.617654] Key type lgssc unregistered [ 1839.747420] LNet: 108222:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1839.750323] LNetError: 108222:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1839.758343] LNet: Removed LNI 192.168.201.103@tcp [ 1840.053472] Key type .llcrypt unregistered [ 1840.054426] Key type ._llcrypt unregistered [ 1851.117377] Key type ._llcrypt registered [ 1851.118809] Key type .llcrypt registered [ 1851.183761] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 1852.189860] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1852.202799] alg: No test for adler32 (adler32-zlib) [ 1853.174259] Lustre: Lustre: Build Version: 2.17.50_193_g407ab0b [ 1853.360363] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 1854.993799] Key type lgssc registered [ 1855.933641] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1861.467799] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1861.482612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1862.768018] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1865.690483] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1872.366300] LustreError: 109406:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1872.381805] LustreError: 109406:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1872.885571] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1873.126193] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1876.282277] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1878.928233] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1880.374762] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1883.385801] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1883.539816] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1886.145389] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1887.656312] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 1888.478514] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1903.073942] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1903.077499] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1903.084121] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1903.086266] Lustre: Skipped 1 previous similar message [ 1907.357310] Lustre: server umount lustre-OST0000 complete [ 1909.216828] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1909.219661] LustreError: Skipped 1 previous similar message [ 1909.221843] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1909.227193] Lustre: Skipped 1 previous similar message [ 1909.229678] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1913.313123] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1913.313564] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1918.434099] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1918.436855] Lustre: Skipped 2 previous similar messages [ 1923.040174] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1923.114737] Lustre: server umount lustre-MDT0000 complete [ 1923.554120] LustreError: 109401:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1923.564095] LustreError: 109401:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1924.727959] LustreError: 109387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772680162 with bad export cookie 2148436125671402535 [ 1924.728731] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1924.738356] LustreError: 109387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1924.875618] Lustre: server umount lustre-MDT0001 complete [ 1930.282340] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 1931.684194] Key type lgssc unregistered [ 1931.863412] LNet: 112607:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1931.867679] LNetError: 112607:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1931.874764] LNet: Removed LNI 192.168.201.103@tcp [ 1932.228380] Key type .llcrypt unregistered [ 1932.229806] Key type ._llcrypt unregistered [ 1943.101849] Key type ._llcrypt registered [ 1943.103462] Key type .llcrypt registered [ 1943.153097] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 1943.534880] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1943.542559] alg: No test for adler32 (adler32-zlib) [ 1944.443797] Lustre: Lustre: Build Version: 2.17.50_193_g407ab0b [ 1944.566656] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 1946.184133] Key type lgssc registered [ 1946.694097] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1950.265143] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1950.276198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1951.482179] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1953.114440] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1956.699169] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1956.913600] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1958.579126] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1960.650523] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1962.185551] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1964.802909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1964.943566] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1967.454910] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1970.156453] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1977.259898] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 1984.864194] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 1986.626485] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 1986.634127] Lustre: Skipped 2 previous similar messages [ 1988.959166] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 1988.963512] Lustre: Skipped 2 previous similar messages [ 1990.154651] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 1990.170196] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2000.922331] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.201.3@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 2000.933570] Lustre: Skipped 2 previous similar messages [ 2006.541121] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.201.3@tcp 192.168.0.[2,10]@tcp [ 2009.178172] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.201.3/24@tcp in log lustre-MDT0000 [ 2009.184560] Lustre: Skipped 2 previous similar messages [ 2014.234684] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.3/24@tcp [ 2014.238151] Lustre: Skipped 1 previous similar message [ 2018.276269] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2018.276752] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2018.280204] LustreError: Skipped 1 previous similar message [ 2018.283109] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2018.283782] Lustre: Skipped 1 previous similar message [ 2019.525777] Lustre: server umount lustre-OST0000 complete [ 2023.394209] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2023.394667] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2023.399388] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2023.404668] Lustre: Skipped 1 previous similar message [ 2023.407238] Lustre: Skipped 1 previous similar message [ 2027.768403] Lustre: server umount lustre-MDT0000 complete [ 2028.513733] LustreError: 117036:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2028.522534] LustreError: 117036:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2029.635731] LustreError: 113966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772680267 with bad export cookie 13417650698889773598 [ 2029.640475] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2029.644640] LustreError: 113966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2029.820550] Lustre: server umount lustre-MDT0001 complete [ 2034.915453] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 2036.229433] Key type lgssc unregistered [ 2036.382248] LNet: 118631:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2036.385298] LNetError: 118631:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2036.397906] LNet: Removed LNI 192.168.201.103@tcp [ 2036.825251] Key type .llcrypt unregistered [ 2036.826443] Key type ._llcrypt unregistered [ 2049.705891] Key type ._llcrypt registered [ 2049.707810] Key type .llcrypt registered [ 2049.753091] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 2050.200174] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2050.233658] alg: No test for adler32 (adler32-zlib) [ 2051.107918] Lustre: Lustre: Build Version: 2.17.50_193_g407ab0b [ 2051.209426] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 2052.808169] Key type lgssc registered [ 2053.316616] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2056.483464] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2056.490778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2057.678160] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 2057.685555] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2057.691450] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.201.3/24@tcp [ 2059.209168] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2062.218127] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2062.321163] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2062.330099] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.3/24@tcp [ 2062.347292] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2063.749697] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2065.796435] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2067.224291] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2070.126415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2070.287528] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2072.744359] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2075.011155] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2083.625865] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 2093.025355] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2093.027260] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2093.030867] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2094.049639] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2094.049880] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2094.052037] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2095.185924] Lustre: server umount lustre-OST0000 complete [ 2098.144913] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2098.148196] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2098.153610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2098.156483] Lustre: Skipped 1 previous similar message [ 2102.681980] Lustre: server umount lustre-MDT0000 complete [ 2104.152601] LustreError: 120180:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772680341 with bad export cookie 18138350033518973375 [ 2104.155013] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2104.158407] LustreError: 120180:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2104.308763] Lustre: server umount lustre-MDT0001 complete [ 2108.693629] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 2110.027247] Key type lgssc unregistered [ 2110.191525] LNet: 123407:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2110.195404] LNetError: 123407:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2110.203958] LNet: Removed LNI 192.168.201.103@tcp [ 2110.565891] Key type .llcrypt unregistered [ 2110.567310] Key type ._llcrypt unregistered [ 2119.939258] Key type ._llcrypt registered [ 2119.940723] Key type .llcrypt registered [ 2119.983438] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 2126.125411] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 2126.557224] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2126.565351] alg: No test for adler32 (adler32-zlib) [ 2127.412464] Lustre: Lustre: Build Version: 2.17.50_193_g407ab0b [ 2127.502247] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 2129.088188] Key type lgssc registered [ 2129.433547] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2140.592697] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2140.598795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2141.688589] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2141.701126] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2141.750106] Lustre: lustre-MDT0000: new disk, initializing [ 2141.776606] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2141.784724] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2143.096888] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2147.139910] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2147.183445] Lustre: 126745:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2147.206363] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2147.208248] Lustre: Skipped 1 previous similar message [ 2147.246416] Lustre: lustre-MDT0001: new disk, initializing [ 2147.278343] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2147.291415] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2147.294637] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2148.640215] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2151.772537] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2152.952337] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2155.277398] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2155.384959] Lustre: lustre-OST0000: new disk, initializing [ 2155.388380] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2155.420890] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2157.334353] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2157.432726] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2157.437358] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2157.460898] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2160.277707] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2161.503755] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2161.574176] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2162.794341] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2162.867682] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2167.778978] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2167.779635] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2167.780402] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2167.786231] Lustre: Skipped 1 previous similar message [ 2169.614876] Lustre: server umount lustre-OST0000 complete [ 2172.897965] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2172.898040] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2172.898531] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2172.898535] Lustre: Skipped 1 previous similar message [ 2172.901804] Lustre: Skipped 1 previous similar message [ 2172.908632] LustreError: Skipped 1 previous similar message [ 2178.017058] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2178.018950] Lustre: Skipped 2 previous similar messages [ 2183.137067] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2183.138675] Lustre: Skipped 1 previous similar message [ 2185.184146] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2185.248988] Lustre: server umount lustre-MDT0000 complete [ 2186.494285] LustreError: 126735:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772680424 with bad export cookie 6149779897479407201 [ 2186.497850] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2186.498045] LustreError: 126735:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2186.623284] Lustre: server umount lustre-MDT0001 complete [ 2189.506223] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2189.684441] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2191.042324] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2193.812315] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2193.942983] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2195.314158] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2197.206630] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2198.536704] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2201.011384] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2203.126081] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2204.850158] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2206.158739] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2215.383426] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 9 sec [ 2216.598356] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2216.665768] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2220.939911] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2220.991899] Lustre: lustre-OST0001: new disk, initializing [ 2220.993382] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2221.021272] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2221.023134] Lustre: Skipped 1 previous similar message [ 2222.785515] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2226.111073] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2230.267610] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2230.270100] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2230.297389] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2231.285082] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 5 sec [ 2232.619254] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2232.693492] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2233.944644] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff964087611800.ost_server_uuid 50 [ 2234.401796] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff964087611800.ost_server_uuid in FULL state after 0 sec [ 2240.484404] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2240.485630] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2240.488729] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2240.493583] Lustre: Skipped 1 previous similar message [ 2242.109248] Lustre: server umount lustre-OST0001 complete [ 2244.065231] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2244.068401] LustreError: Skipped 1 previous similar message [ 2244.070024] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2244.074405] Lustre: Skipped 1 previous similar message [ 2245.602130] LustreError: 132143:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2245.605609] LustreError: 132143:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2249.419820] Lustre: server umount lustre-OST0000 complete [ 2250.721162] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2250.724637] Lustre: Skipped 1 previous similar message [ 2250.726402] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2250.727956] Lustre: Skipped 3 previous similar messages [ 2255.328807] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2255.332979] LustreError: Skipped 1 previous similar message [ 2256.786915] Lustre: server umount lustre-MDT0000 complete [ 2257.983990] LustreError: 130534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772680495 with bad export cookie 6149779897479408069 [ 2257.986598] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2257.987622] LustreError: 130534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2258.104840] Lustre: server umount lustre-MDT0001 complete [ 2264.280532] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 2265.267673] Key type lgssc unregistered [ 2265.395490] LNet: 136486:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2265.397759] LNetError: 136486:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2265.405574] LNet: Removed LNI 192.168.201.103@tcp [ 2265.699785] Key type .llcrypt unregistered [ 2265.700732] Key type ._llcrypt unregistered [ 2277.372702] Key type ._llcrypt registered [ 2277.373557] Key type .llcrypt registered [ 2277.408541] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 2283.128726] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 2283.424566] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2283.432397] alg: No test for adler32 (adler32-zlib) [ 2284.271220] Lustre: Lustre: Build Version: 2.17.50_193_g407ab0b [ 2284.344886] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 2285.920137] Key type lgssc registered [ 2286.239742] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2296.444434] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2296.450174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2297.530107] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2297.540242] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2297.573804] Lustre: lustre-MDT0000: new disk, initializing [ 2297.596834] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2297.602255] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2298.776932] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2302.698182] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2302.739053] Lustre: 140576:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2302.762070] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2302.764337] Lustre: Skipped 1 previous similar message [ 2302.801245] Lustre: lustre-MDT0001: new disk, initializing [ 2302.834307] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2302.846540] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2302.850363] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2304.046676] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2307.230193] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2308.392303] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2310.540494] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2310.632471] Lustre: lustre-OST0000: new disk, initializing [ 2310.634313] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2310.659377] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2312.382796] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2313.398600] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2313.401138] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2313.432350] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2315.385112] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2317.249479] Lustre: Failing over lustre-OST0000 [ 2317.315104] Lustre: server umount lustre-OST0000 complete [ 2330.037420] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2330.137781] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2330.142678] Lustre: Skipped 1 previous similar message [ 2330.148314] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2330.153896] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2331.071381] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2331.883455] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2333.179386] Lustre: Failing over lustre-MDT0000 [ 2333.268808] Lustre: server umount lustre-MDT0000 complete [ 2333.664715] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2333.666618] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2334.688165] Lustre: 138803:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772680556/real 1772680556] req@0000000079297f43 x1858790251061504/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1772680572 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2335.201871] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2335.205209] Lustre: Skipped 1 previous similar message [ 2339.808186] Lustre: 138803:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772680561/real 1772680561] req@000000002b941754 x1858790251063168/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1772680577 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2340.320956] LustreError: 140586:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2340.325541] LustreError: 140586:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 2341.232924] LustreError: 141706:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2344.416102] Lustre: 138804:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772680566/real 1772680566] req@00000000f8cc665a x1858790251064320/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1772680582 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2345.441601] LustreError: 140585:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2345.446478] LustreError: 140585:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2346.077662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2346.112779] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2346.115921] Lustre: 140578:0:(mgc_request.c:1911:mgc_process_log()) MGC192.168.201.103@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 2346.182424] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 2346.198700] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2346.199115] Lustre: lustre-OST0000: Recovery over after 0:15, of 3 clients 3 recovered and 0 were evicted. [ 2346.201502] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2346.222442] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2346.352887] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2347.433921] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2351.590507] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2351.592949] Lustre: Skipped 1 previous similar message [ 2351.596974] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2351.612850] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2356.708996] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2356.709478] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2356.714092] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2356.717432] Lustre: Skipped 1 previous similar message [ 2358.737630] Lustre: server umount lustre-OST0000 complete [ 2361.825591] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2361.826532] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2361.827080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2361.827923] LustreError: Skipped 1 previous similar message [ 2361.834771] Lustre: Skipped 2 previous similar messages [ 2366.232651] Lustre: server umount lustre-MDT0000 complete [ 2366.945908] LustreError: 140586:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2366.951658] LustreError: 140586:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2367.515175] LustreError: 140567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772680605 with bad export cookie 15953569815554540975 [ 2367.518590] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2367.518749] LustreError: 140567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2367.637670] Lustre: server umount lustre-MDT0001 complete [ 2371.151843] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 2372.072689] Key type lgssc unregistered [ 2372.186334] LNet: 145803:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2372.188198] LNetError: 145803:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2372.194323] LNet: Removed LNI 192.168.201.103@tcp [ 2372.457728] Key type .llcrypt unregistered [ 2372.458932] Key type ._llcrypt unregistered [ 2391.944358] Key type ._llcrypt registered [ 2391.945675] Key type .llcrypt registered [ 2391.984324] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 2392.437217] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2392.456531] alg: No test for adler32 (adler32-zlib) [ 2393.310871] Lustre: Lustre: Build Version: 2.17.50_193_g407ab0b [ 2393.398788] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 2394.984133] Key type lgssc registered [ 2395.353338] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2398.007350] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2398.013789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2399.175859] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2400.463690] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2403.609096] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2405.589568] LustreError: 147330:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2405.592684] LustreError: 147330:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2405.595245] LustreError: 147330:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 6, retries 0, failed: rc = -5 [ 2405.707972] Lustre: server umount lustre-MDT0000 complete [ 2410.532829] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 2413.526378] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 2424.877086] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2424.971678] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2424.984473] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2425.023197] Lustre: lustre-MDT0000: new disk, initializing [ 2425.046523] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2425.052638] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2426.300763] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2430.375802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2430.418274] Lustre: 150481:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2430.443686] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2430.445312] Lustre: Skipped 1 previous similar message [ 2430.484954] Lustre: lustre-MDT0001: new disk, initializing [ 2430.515436] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2430.526696] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2430.530562] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2431.792767] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2434.856986] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2436.041589] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2438.244208] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2438.359814] Lustre: lustre-OST0000: new disk, initializing [ 2438.362387] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2438.399054] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2440.248913] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2440.250703] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2440.253662] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2440.292252] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2443.302952] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2445.075295] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 2445.610365] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 22:18:03 (1772680683) [ 2511.840526] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2511.840772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2511.845235] Lustre: Skipped 2 previous similar messages [ 2511.848865] Lustre: Skipped 2 previous similar messages [ 2514.080760] Lustre: server umount lustre-MDT0000 complete [ 2515.250087] LustreError: 151258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772680753 with bad export cookie 13106288316502534539 [ 2515.250943] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2515.253825] LustreError: 151258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2516.960428] LustreError: 150487:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2516.960477] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2516.967594] LustreError: 150487:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2516.971091] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2520.992448] LustreError: 150486:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2520.992460] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2521.444423] Lustre: server umount lustre-MDT0001 complete [ 2528.774602] Lustre: server umount lustre-OST0000 complete [ 2531.608674] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 2534.523583] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 2545.990186] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2546.078239] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2546.089682] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2546.129960] Lustre: lustre-MDT0000: new disk, initializing [ 2546.154134] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2546.161367] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2547.415323] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2551.441445] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2551.480393] Lustre: 156701:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2551.503139] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2551.504691] Lustre: Skipped 1 previous similar message [ 2551.540769] Lustre: lustre-MDT0001: new disk, initializing [ 2551.586397] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2551.589949] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2552.855972] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2555.873705] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2557.019896] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2559.162267] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2559.289897] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2559.291775] Lustre: Skipped 1 previous similar message [ 2561.077919] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2561.911052] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2561.913912] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2561.946858] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2564.110761] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2565.398416] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2565.465391] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2566.789988] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2566.884369] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2572.259345] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2572.259934] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2572.261777] LustreError: Skipped 1 previous similar message [ 2572.265843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2572.268563] Lustre: Skipped 1 previous similar message [ 2573.648372] Lustre: server umount lustre-OST0000 complete [ 2577.378105] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2577.378112] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2577.378636] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2577.385016] Lustre: Skipped 2 previous similar messages [ 2581.078394] Lustre: server umount lustre-MDT0000 complete [ 2582.322364] LustreError: 156694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772680820 with bad export cookie 13106288316502693355 [ 2582.325528] LustreError: 156694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2582.325794] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2582.435300] Lustre: server umount lustre-MDT0001 complete [ 2585.401149] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 22:20:23 (1772680823) [ 2590.546426] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 2593.470267] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 2605.070511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2605.156549] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2605.167164] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2605.168727] Lustre: Skipped 1 previous similar message [ 2605.206958] Lustre: lustre-MDT0000: new disk, initializing [ 2605.208292] Lustre: Skipped 1 previous similar message [ 2605.233162] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2605.241518] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2606.667015] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2610.823218] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2610.869928] Lustre: 163016:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2610.905579] LustreError: 163026:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2610.999698] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2612.400721] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2615.415040] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2616.578684] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2618.793702] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2620.283736] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2620.322205] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2620.750297] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2623.749195] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2630.627918] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2630.628412] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2630.637026] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2630.638916] Lustre: Skipped 2 previous similar messages [ 2631.899207] Lustre: server umount lustre-OST0000 complete [ 2639.197378] Lustre: server umount lustre-MDT0000 complete [ 2640.522373] LustreError: 163007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772680878 with bad export cookie 13106288316502694223 [ 2640.526249] LustreError: 163007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2640.526414] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2640.650048] Lustre: server umount lustre-MDT0001 complete [ 2643.392149] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 22:21:21 (1772680881) [ 2648.559916] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 2651.400871] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 2662.463699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2662.552107] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2662.563632] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2662.565152] Lustre: Skipped 3 previous similar messages [ 2662.604228] Lustre: lustre-MDT0000: new disk, initializing [ 2662.605582] Lustre: Skipped 2 previous similar messages [ 2662.630222] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2662.632882] Lustre: Skipped 2 previous similar messages [ 2662.638496] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2662.641234] Lustre: Skipped 2 previous similar messages [ 2663.897795] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2668.011667] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2668.057156] Lustre: 168930:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2668.161824] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2669.468637] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2672.528579] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2673.693321] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2675.878712] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2677.278547] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2677.826187] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2680.855561] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2687.457507] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2687.459246] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2687.460652] LustreError: Skipped 2 previous similar messages [ 2687.463592] Lustre: Skipped 3 previous similar messages [ 2687.464230] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2687.468274] Lustre: Skipped 3 previous similar messages [ 2688.781561] Lustre: server umount lustre-OST0000 complete [ 2704.352120] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2704.422230] Lustre: server umount lustre-MDT0000 complete [ 2705.707384] LustreError: 169713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772680943 with bad export cookie 13106288316502695280 [ 2705.709764] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2705.710296] LustreError: 169713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2705.820688] Lustre: server umount lustre-MDT0001 complete [ 2709.176356] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 2710.222270] Key type lgssc unregistered [ 2710.353406] LNet: 172404:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2710.356215] LNetError: 172404:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2710.362426] LNet: Removed LNI 192.168.201.103@tcp [ 2710.692822] Key type .llcrypt unregistered [ 2710.693907] Key type ._llcrypt unregistered [ 2719.302892] Key type ._llcrypt registered [ 2719.303927] Key type .llcrypt registered [ 2719.339147] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 2719.677685] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2719.712258] alg: No test for adler32 (adler32-zlib) [ 2720.561760] Lustre: Lustre: Build Version: 2.17.50_193_g407ab0b [ 2720.643290] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 2722.232190] Key type lgssc registered [ 2722.637557] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2725.382332] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2725.390152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2726.584417] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2727.960871] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2730.806219] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2730.931942] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2732.321573] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2733.997863] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2735.162800] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2737.333301] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2737.454339] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2739.261198] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2740.962826] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2751.783488] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2771.425130] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2771.427777] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2771.431374] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2772.450861] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2772.452191] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2772.452386] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2772.460091] Lustre: Skipped 1 previous similar message [ 2774.108236] Lustre: server umount lustre-OST0000 complete [ 2777.056747] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2777.058943] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2777.062804] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2781.460624] Lustre: server umount lustre-MDT0000 complete [ 2782.623121] LustreError: 175949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772681020 with bad export cookie 12722014582619253272 [ 2782.625815] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2782.627255] LustreError: 175949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2782.747513] Lustre: server umount lustre-MDT0001 complete [ 2786.179825] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 2787.183745] Key type lgssc unregistered [ 2787.303358] LNet: 176983:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2787.306190] LNetError: 176983:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2787.315336] LNet: Removed LNI 192.168.201.103@tcp [ 2787.604453] Key type .llcrypt unregistered [ 2787.605546] Key type ._llcrypt unregistered [ 2796.190473] Key type ._llcrypt registered [ 2796.191375] Key type .llcrypt registered [ 2796.227968] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 2796.654553] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2796.668921] alg: No test for adler32 (adler32-zlib) [ 2797.519324] Lustre: Lustre: Build Version: 2.17.50_193_g407ab0b [ 2797.614736] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 2799.200171] Key type lgssc registered [ 2799.532587] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2802.145579] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2802.154686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2803.318629] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2804.655738] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2807.394962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2807.519335] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2808.803226] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2810.505033] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2811.687230] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2813.843031] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2813.923114] LustreError: 179951:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2813.928555] LustreError: 179951:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2813.991417] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2815.951859] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2817.669559] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2818.087500] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 2828.260954] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2828.261380] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2828.263027] LustreError: Skipped 1 previous similar message [ 2828.263545] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2828.270418] Lustre: Skipped 1 previous similar message [ 2829.176845] Lustre: lustre-OST0000: Not available for connect from 192.168.201.3@tcp (stopping) [ 2829.179939] Lustre: Skipped 1 previous similar message [ 2832.140817] Lustre: server umount lustre-OST0000 complete [ 2833.868140] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff964089c61000.ost_server_uuid 50 [ 2834.351860] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff964089c61000.ost_server_uuid in DISCONN state after 0 sec [ 2834.919466] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 2848.738069] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2848.738089] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2848.739110] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2848.741108] Lustre: Skipped 1 previous similar message [ 2851.029565] Lustre: server umount lustre-MDT0000 complete [ 2852.236073] LustreError: 179076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772681090 with bad export cookie 17319960819170671278 [ 2852.239052] LustreError: 179076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2852.239937] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2852.345115] Lustre: server umount lustre-MDT0001 complete [ 2854.943736] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 22:24:53 (1772681093) [ 2857.078367] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2857.227247] LustreError: 181867:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2857.247971] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2858.495474] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2861.181646] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2861.296934] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2862.489972] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2864.173256] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2865.293067] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2867.454959] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2868.583050] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 2869.375557] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2871.060972] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2873.291036] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2873.362724] Lustre: lustre-OST0001: new disk, initializing [ 2873.364442] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2873.396522] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2873.398429] Lustre: Skipped 1 previous similar message [ 2875.209155] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2878.260296] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2882.017323] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2882.019370] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2882.022914] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2882.024473] Lustre: Skipped 3 previous similar messages [ 2883.576090] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2883.581109] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2883.611059] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2885.178311] Lustre: server umount lustre-OST0000 complete [ 2887.077350] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2887.142839] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 2888.406600] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2888.481254] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 2888.562146] LustreError: 184070:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2893.682197] LustreError: 183462:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2893.687170] LustreError: 183462:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2898.915816] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2898.918174] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2898.918915] LustreError: 183462:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2898.918929] LustreError: 183462:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2898.919807] LustreError: Skipped 1 previous similar message [ 2898.922819] Lustre: Skipped 1 previous similar message [ 2898.923344] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2898.938024] Lustre: Skipped 4 previous similar messages [ 2904.539309] Lustre: server umount lustre-OST0001 complete [ 2907.616855] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2907.619144] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2907.622427] Lustre: Skipped 1 previous similar message [ 2907.623942] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2907.625777] Lustre: Skipped 2 previous similar messages [ 2911.918035] Lustre: server umount lustre-MDT0000 complete [ 2913.214378] LustreError: 181848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772681151 with bad export cookie 17319960819170672342 [ 2913.219324] LustreError: 181848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2913.221882] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2913.363635] Lustre: server umount lustre-MDT0001 complete [ 2920.722042] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 22:25:59 (1772681159) [ 2922.893151] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2922.995203] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2923.007429] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2923.081823] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2924.434715] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2927.293391] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2927.327861] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2927.341881] Lustre: 187635:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2928.645135] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2930.375340] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2931.622551] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2934.023205] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2934.099620] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2934.647964] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 2935.973244] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2937.698347] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2939.944138] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2939.973904] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2941.841797] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2943.533102] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2944.993232] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2944.993549] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2944.997052] LustreError: Skipped 1 previous similar message [ 2945.001106] Lustre: Skipped 1 previous similar message [ 2945.004061] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2945.007576] Lustre: Skipped 3 previous similar messages [ 2950.462717] Lustre: server umount lustre-OST0000 complete [ 2953.698927] LustreError: 190820:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2953.704425] LustreError: 190820:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2963.940374] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2963.940938] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2963.946314] LustreError: Skipped 1 previous similar message [ 2967.118415] Lustre: server umount lustre-OST0001 complete [ 2974.492556] Lustre: server umount lustre-MDT0000 complete [ 2975.717604] LustreError: 187625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772681213 with bad export cookie 17319960819170673700 [ 2975.720204] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2975.720841] LustreError: 187625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2975.852800] Lustre: server umount lustre-MDT0001 complete [ 2983.245052] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 22:27:01 (1772681221) [ 2988.149726] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 2990.980855] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 3002.296761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3002.382110] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3002.391485] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3002.427270] Lustre: lustre-MDT0000: new disk, initializing [ 3002.450327] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3002.452403] Lustre: Skipped 3 previous similar messages [ 3002.457845] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3003.715489] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3007.768627] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3007.808938] Lustre: 195207:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3007.831160] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3007.832783] Lustre: Skipped 1 previous similar message [ 3007.867842] Lustre: lustre-MDT0001: new disk, initializing [ 3007.906757] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3007.910000] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3009.203264] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3012.271965] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3013.406912] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3015.669937] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3015.776206] Lustre: lustre-OST0000: new disk, initializing [ 3015.778611] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3017.581460] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3018.940422] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3018.943719] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3018.976943] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3020.554564] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3021.956110] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3022.024982] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3023.244741] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3023.310317] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3024.355804] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3024.356195] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3024.356757] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3024.356762] Lustre: Skipped 10 previous similar messages [ 3024.357906] LustreError: Skipped 2 previous similar messages [ 3024.360703] Lustre: Skipped 4 previous similar messages [ 3030.047115] Lustre: server umount lustre-OST0000 complete [ 3037.469403] Lustre: server umount lustre-MDT0000 complete [ 3038.671031] LustreError: 195984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772681276 with bad export cookie 17319960819170675247 [ 3038.673951] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3038.675582] LustreError: 195984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3038.777416] Lustre: server umount lustre-MDT0001 complete [ 3041.769488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3041.927149] LustreError: 199016:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3041.932297] LustreError: 199016:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 3043.198665] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3045.896951] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3047.263492] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3048.962647] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3050.093401] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3052.244779] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3054.171657] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3055.937534] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3057.287457] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3067.577133] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 10 sec [ 3068.808307] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3068.879647] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3075.660656] Lustre: server umount lustre-OST0000 complete [ 3077.506061] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3078.812962] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3102.301458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3102.413864] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3102.416398] Lustre: Skipped 5 previous similar messages [ 3104.247489] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3105.980706] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3106.497203] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3109.448626] Lustre: server umount lustre-OST0000 complete [ 3110.676073] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3110.676739] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3110.682971] Lustre: Skipped 5 previous similar messages [ 3110.685124] Lustre: Skipped 9 previous similar messages [ 3116.831102] Lustre: server umount lustre-MDT0000 complete [ 3118.127882] LustreError: 198997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772681355 with bad export cookie 17319960819170676115 [ 3118.131452] LustreError: 198997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3118.132354] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3118.250023] Lustre: server umount lustre-MDT0001 complete [ 3120.815660] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 22:29:19 (1772681359) [ 3122.980642] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3123.151442] LustreError: 204529:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3124.444673] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3127.188205] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3128.516907] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3130.213257] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3131.299839] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3133.583293] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3135.503323] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3137.322644] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3138.707798] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3150.004372] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 11 sec [ 3151.301714] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3151.382648] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3153.556703] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3153.618701] Lustre: lustre-OST0001: new disk, initializing [ 3153.620493] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3155.418125] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3158.360170] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3159.582128] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3160.576374] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3160.581489] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3160.605331] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3160.664822] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 3161.859363] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3161.924818] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3165.665955] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3165.669278] LustreError: Skipped 3 previous similar messages [ 3168.573699] Lustre: server umount lustre-OST0001 complete [ 3170.413969] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3170.482198] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3171.703294] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3171.774061] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3187.570452] LustreError: 208430:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3187.578412] LustreError: 208430:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 12 previous similar messages [ 3195.271013] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3197.111323] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3198.929151] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3199.450125] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3206.143246] Lustre: server umount lustre-OST0001 complete [ 3224.202719] Lustre: server umount lustre-OST0000 complete [ 3231.709640] Lustre: server umount lustre-MDT0000 complete [ 3232.930163] LustreError: 204510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772681470 with bad export cookie 17319960819170677046 [ 3232.934131] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3232.934865] LustreError: 204510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3239.088529] Lustre: server umount lustre-MDT0001 complete [ 3246.480045] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 22:31:24 (1772681484) [ 3248.638071] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3248.726571] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3248.737737] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3248.817525] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3248.819204] Lustre: Skipped 5 previous similar messages [ 3250.057524] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3252.765804] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3252.794541] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3252.805943] Lustre: 212550:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3254.126674] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3255.954317] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3257.109071] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3259.471019] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3259.538658] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3261.661780] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3263.516898] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3272.756899] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3272.796180] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3274.987375] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3276.804577] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3277.314068] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3278.576444] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3278.646979] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3279.931989] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3279.999650] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3281.173249] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff964089c46800.ost_server_uuid 50 [ 3281.705365] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff964089c46800.ost_server_uuid in FULL state after 0 sec [ 3282.055307] Lustre: Permanently deactivating lustre-OST0001 [ 3282.061571] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3282.930907] Lustre: Permanently reactivating lustre-OST0001 [ 3284.967128] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3284.971088] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3284.971553] Lustre: Skipped 7 previous similar messages [ 3284.973176] Lustre: Skipped 18 previous similar messages [ 3290.048478] Lustre: server umount lustre-OST0001 complete [ 3297.431496] Lustre: server umount lustre-OST0000 complete [ 3299.297244] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3299.300953] LustreError: Skipped 3 previous similar messages [ 3305.075924] Lustre: server umount lustre-MDT0000 complete [ 3306.341768] LustreError: 212541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772681544 with bad export cookie 17319960819170678306 [ 3306.345932] LustreError: 212541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3306.347346] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3306.471792] Lustre: server umount lustre-MDT0001 complete [ 3314.529535] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 22:32:32 (1772681552) [ 3317.240372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3317.336534] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3317.347158] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3317.351170] Lustre: Skipped 5 previous similar messages [ 3318.728816] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3321.991594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3322.033750] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3322.044759] Lustre: 218648:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3323.456737] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3325.220719] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3326.354362] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3328.615847] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3328.694140] Lustre: Permanently deactivating lustre-OST0000 [ 3328.698273] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 3330.602784] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3332.351156] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3334.129280] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3334.133080] LustreError: 221100:0:(osp_precreate.c:651:osp_precreate_send()) lustre-OST0000-osc-MDT0001: can't precreate: rc = -5 [ 3334.136102] LustreError: 221100:0:(osp_precreate.c:1342:osp_precreate_thread()) lustre-OST0000-osc-MDT0001: cannot precreate objects: rc = -5 [ 3334.497123] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3336.365042] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3338.103196] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3342.377933] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 3344.136163] Lustre: Permanently reactivating lustre-OST0000 [ 3351.535422] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 3351.539315] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3351.542251] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3351.554897] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 3351.561987] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3352.961324] Lustre: server umount lustre-OST0001 complete [ 3357.668891] LustreError: 221101:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3357.676016] LustreError: 221101:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 16 previous similar messages [ 3360.266198] Lustre: server umount lustre-OST0000 complete [ 3361.651769] Lustre: server umount lustre-MDT0000 complete [ 3362.950335] LustreError: 218639:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772681600 with bad export cookie 17319960819170679944 [ 3362.953626] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3362.953893] LustreError: 218639:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3369.141877] Lustre: server umount lustre-MDT0001 complete [ 3372.676680] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 3373.699329] Key type lgssc unregistered [ 3373.825399] LNet: 223026:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3373.827556] LNetError: 223026:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3373.833444] LNet: Removed LNI 192.168.201.103@tcp [ 3374.129449] Key type .llcrypt unregistered [ 3374.130537] Key type ._llcrypt unregistered [ 3382.594855] Key type ._llcrypt registered [ 3382.595810] Key type .llcrypt registered [ 3382.627745] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 3382.926700] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3382.969021] alg: No test for adler32 (adler32-zlib) [ 3383.811035] Lustre: Lustre: Build Version: 2.17.50_193_g407ab0b [ 3383.902230] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 3385.480139] Key type lgssc registered [ 3385.827711] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3388.968290] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3388.974303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3390.138463] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3391.352091] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3394.107214] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3394.145869] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3394.157468] Lustre: 224442:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3394.163547] Lustre: Permanently deactivating lustre-MDT0001 [ 3394.169320] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3394.294160] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3395.566330] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3397.358921] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3398.498359] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3398.636345] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3398.638396] LustreError: 224484:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3398.643081] LustreError: 224484:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3398.646060] LustreError: 224484:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 1, failed: rc = -108 [ 3400.632640] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3400.748773] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3402.515795] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3404.288061] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3404.838332] LustreError: 226067:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3404.844028] LustreError: 226067:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3406.521654] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3406.606682] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3406.819549] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 3408.488408] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3410.269020] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3415.016412] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 3415.017312] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 3416.312418] Lustre: Permanently reactivating lustre-MDT0001 [ 3416.318359] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 3416.320570] Lustre: Skipped 2 previous similar messages [ 3417.719293] Lustre: lustre-MDT0001: Client 2a24949f-c5f1-4c4e-9d13-e63aa5b46771 (at 192.168.201.3@tcp) reconnecting [ 3419.503678] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3420.012250] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3421.611683] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 3421.686910] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 3423.030656] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 3425.139703] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 2 sec [ 3425.936098] Lustre: Permanently deactivating lustre-MDT0001 [ 3425.944129] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3425.946880] Lustre: Skipped 2 previous similar messages [ 3432.961839] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3440.611394] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3440.611800] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3440.612443] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3440.615530] Lustre: Skipped 1 previous similar message [ 3443.987682] Lustre: server umount lustre-MDT0000 complete [ 3445.164198] LustreError: 227553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772681683 with bad export cookie 1373189890602704196 [ 3445.166995] LustreError: 227553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3445.167236] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3445.272941] Lustre: server umount lustre-MDT0001 complete [ 3452.539515] Lustre: server umount lustre-OST0000 complete [ 3456.074407] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 22:34:54 (1772681694) [ 3459.431634] Lustre: server umount lustre-OST0001 complete [ 3461.270356] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 3463.947725] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 3475.747223] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3475.844213] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3475.848940] Lustre: Skipped 2 previous similar messages [ 3475.857977] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3475.898953] Lustre: lustre-MDT0000: new disk, initializing [ 3475.924725] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3475.930559] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3477.299889] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3481.267214] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3481.307298] Lustre: 232123:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3481.331524] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3481.333471] Lustre: Skipped 1 previous similar message [ 3481.375312] Lustre: lustre-MDT0001: new disk, initializing [ 3481.416624] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3481.419757] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3482.646800] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3485.692858] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3486.869866] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3489.067089] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3489.159771] Lustre: lustre-OST0000: new disk, initializing [ 3489.161574] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3489.190469] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3489.192202] Lustre: Skipped 1 previous similar message [ 3489.848759] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3489.851519] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3489.882367] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3490.969304] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3493.943564] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3494.539803] LustreError: 232130:0:(mdt_handler.c:3252:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3498.735333] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3498.801218] Lustre: lustre-OST0001: new disk, initializing [ 3498.802961] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3500.905627] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3504.285647] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3504.632093] LustreError: 232130:0:(mdt_handler.c:3252:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3504.642382] LustreError: 232129:0:(mdt_handler.c:3252:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3504.942285] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3506.685507] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3506.689843] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3506.721705] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3514.736082] LustreError: 232129:0:(mdt_handler.c:3252:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3516.901119] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3516.902099] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3516.903403] LustreError: Skipped 1 previous similar message [ 3516.907509] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3516.911019] Lustre: Skipped 4 previous similar messages [ 3519.358224] Lustre: lustre-OST0001: Not available for connect from 192.168.201.3@tcp (stopping) [ 3521.467282] Lustre: server umount lustre-OST0001 complete [ 3522.020807] LustreError: 233846:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3522.026125] LustreError: 233846:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3527.139812] LustreError: 233847:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3527.144007] LustreError: 233847:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3532.260740] LustreError: 233997:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3532.265554] LustreError: 233997:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3535.840938] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3535.841125] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3535.842833] LustreError: Skipped 1 previous similar message [ 3535.843286] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3535.848466] Lustre: Skipped 2 previous similar messages [ 3535.853617] Lustre: Skipped 1 previous similar message [ 3537.380210] LustreError: 233847:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3537.383824] LustreError: 233847:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3539.534752] Lustre: server umount lustre-OST0000 complete [ 3542.499285] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3542.500254] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3542.503450] Lustre: Skipped 1 previous similar message [ 3542.506478] Lustre: Skipped 2 previous similar messages [ 3552.737676] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3552.739308] Lustre: Skipped 4 previous similar messages [ 3555.296131] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3555.401587] Lustre: server umount lustre-MDT0000 complete [ 3556.727575] LustreError: 234863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772681794 with bad export cookie 1373189890602706765 [ 3556.733150] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3571.168238] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3571.291867] Lustre: server umount lustre-MDT0001 complete [ 3575.530372] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 3576.684487] Key type lgssc unregistered [ 3576.810422] LNet: 236765:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3576.812713] LNetError: 236765:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3576.824437] LNet: Removed LNI 192.168.201.103@tcp [ 3577.136574] Key type .llcrypt unregistered [ 3577.137658] Key type ._llcrypt unregistered [ 3592.104397] Key type ._llcrypt registered [ 3592.105353] Key type .llcrypt registered [ 3592.140660] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 3592.520232] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3592.549280] alg: No test for adler32 (adler32-zlib) [ 3593.398598] Lustre: Lustre: Build Version: 2.17.50_193_g407ab0b [ 3593.487216] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 3595.072114] Key type lgssc registered [ 3595.391578] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3597.742116] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3597.749358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3598.844124] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3598.852104] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3598.920985] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3600.133051] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3602.775240] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3602.811494] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3602.824941] Lustre: 238933:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3602.916830] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3604.161385] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3605.906707] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3607.103424] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3609.308929] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3609.390620] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3609.465031] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3611.329563] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3613.135964] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3614.243329] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 3660.259295] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3660.260558] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3660.263930] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3660.268592] Lustre: Skipped 1 previous similar message [ 3663.962246] Lustre: server umount lustre-OST0000 complete [ 3665.528557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 3669.576139] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3669.704826] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3671.583183] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3673.249369] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3680.997912] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3681.001137] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3681.001219] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3681.006498] Lustre: Skipped 1 previous similar message [ 3706.849605] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3706.850256] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3706.851571] LustreError: Skipped 2 previous similar messages [ 3706.858150] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3706.860117] Lustre: Skipped 2 previous similar messages [ 3711.971857] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3711.974306] Lustre: Skipped 1 previous similar message [ 3712.929793] Lustre: server umount lustre-OST0000 complete [ 3715.552929] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3715.556876] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3715.563187] Lustre: Skipped 1 previous similar message [ 3715.565641] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3720.345266] Lustre: server umount lustre-MDT0000 complete [ 3721.824141] LustreError: 238969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772681959 with bad export cookie 4013273067384923368 [ 3721.827768] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3721.828295] LustreError: 238969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3721.943158] Lustre: server umount lustre-MDT0001 complete [ 3725.818431] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 3726.950725] Key type lgssc unregistered [ 3727.096582] LNet: 243877:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3727.099847] LNetError: 243877:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3727.109526] LNet: Removed LNI 192.168.201.103@tcp [ 3727.480122] Key type .llcrypt unregistered [ 3727.481712] Key type ._llcrypt unregistered [ 3737.703445] Key type ._llcrypt registered [ 3737.704826] Key type .llcrypt registered [ 3737.751891] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 3738.132451] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3738.146339] alg: No test for adler32 (adler32-zlib) [ 3739.031404] Lustre: Lustre: Build Version: 2.17.50_193_g407ab0b [ 3739.129603] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 3740.736325] Key type lgssc registered [ 3741.214622] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3744.432175] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3744.444486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3745.627743] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3747.107524] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3750.513348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3750.679778] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3752.328180] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3754.179733] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3755.383606] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3757.718046] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3757.831521] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3760.089494] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3762.169575] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3771.178745] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 3781.603860] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3781.604212] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3781.606307] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3781.607379] Lustre: Skipped 1 previous similar message [ 3786.129067] Lustre: server umount lustre-OST0000 complete [ 3791.843369] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3791.843368] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3791.844887] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3791.845584] LustreError: Skipped 1 previous similar message [ 3791.848983] Lustre: Skipped 2 previous similar messages [ 3793.695384] Lustre: server umount lustre-MDT0000 complete [ 3794.920181] LustreError: 248202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772682032 with bad export cookie 14934422480569497513 [ 3794.923497] LustreError: 248202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3794.923807] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3795.031800] Lustre: server umount lustre-MDT0001 complete [ 3798.722862] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 3799.739988] Key type lgssc unregistered [ 3799.863384] LNet: 249224:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3799.865723] LNetError: 249224:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3799.875427] LNet: Removed LNI 192.168.201.103@tcp [ 3800.206447] Key type .llcrypt unregistered [ 3800.208437] Key type ._llcrypt unregistered [ 3807.936131] Key type ._llcrypt registered [ 3807.937046] Key type .llcrypt registered [ 3807.972478] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 3808.271363] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3808.279801] alg: No test for adler32 (adler32-zlib) [ 3809.151973] Lustre: Lustre: Build Version: 2.17.50_193_g407ab0b [ 3809.246431] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 3810.832152] Key type lgssc registered [ 3811.183869] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3813.854100] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3813.861899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3815.052469] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3816.405184] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3819.319081] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3819.439732] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3820.840329] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3823.087397] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3824.651772] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3827.218337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3827.356568] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3829.336241] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3831.301782] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3839.669925] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 3850.210295] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3850.211281] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3850.213023] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3850.213282] LustreError: Skipped 1 previous similar message [ 3850.216417] Lustre: Skipped 1 previous similar message [ 3851.609735] Lustre: server umount lustre-OST0000 complete [ 3855.331306] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3855.331386] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3855.333091] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3855.333100] Lustre: Skipped 1 previous similar message [ 3855.338396] Lustre: Skipped 1 previous similar message [ 3859.175181] Lustre: server umount lustre-MDT0000 complete [ 3860.450641] LustreError: 251628:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3860.458051] LustreError: 251628:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3860.487740] LustreError: 252021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772682098 with bad export cookie 8637651676848679082 [ 3860.493148] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3860.497565] LustreError: 252021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3860.615274] Lustre: server umount lustre-MDT0001 complete [ 3864.539809] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 3865.584963] Key type lgssc unregistered [ 3865.741659] LNet: 253749:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3865.745666] LNetError: 253749:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3865.757462] LNet: Removed LNI 192.168.201.103@tcp [ 3866.133753] Key type .llcrypt unregistered [ 3866.135127] Key type ._llcrypt unregistered [ 3875.981391] Key type ._llcrypt registered [ 3875.982453] Key type .llcrypt registered [ 3876.016584] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 3876.348152] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3876.359061] alg: No test for adler32 (adler32-zlib) [ 3877.216554] Lustre: Lustre: Build Version: 2.17.50_193_g407ab0b [ 3877.294148] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 3878.872155] Key type lgssc registered [ 3879.292533] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3881.859293] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3881.865649] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3883.030243] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3884.342752] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3887.586502] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3887.716287] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3888.985439] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3890.828893] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3892.200927] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3894.616957] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3894.725701] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3896.547669] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3898.257389] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3908.073292] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3918.308072] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3918.309569] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3918.310691] LustreError: Skipped 1 previous similar message [ 3918.314624] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3918.316898] Lustre: Skipped 1 previous similar message [ 3922.391807] Lustre: server umount lustre-OST0000 complete [ 3928.547356] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3928.547955] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3928.548934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3928.556381] Lustre: Skipped 2 previous similar messages [ 3929.768154] Lustre: server umount lustre-MDT0000 complete [ 3931.159584] LustreError: 255104:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772682168 with bad export cookie 522476613418186200 [ 3931.164095] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3931.165624] LustreError: 255104:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3931.290229] Lustre: server umount lustre-MDT0001 complete [ 3934.765917] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 3935.839903] Key type lgssc unregistered [ 3935.979334] LNet: 259094:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3935.981705] LNetError: 259094:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3935.988352] LNet: Removed LNI 192.168.201.103@tcp [ 3936.263842] Key type .llcrypt unregistered [ 3936.265059] Key type ._llcrypt unregistered [ 3943.340184] Key type ._llcrypt registered [ 3943.341201] Key type .llcrypt registered [ 3943.374586] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 3943.756642] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3943.807439] alg: No test for adler32 (adler32-zlib) [ 3944.668967] Lustre: Lustre: Build Version: 2.17.50_193_g407ab0b [ 3944.755505] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 3946.336244] Key type lgssc registered [ 3946.727458] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3949.294621] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3949.302402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3950.499074] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3952.014786] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3955.629040] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3955.773157] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3957.173134] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3958.964586] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3960.379561] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3962.715795] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3962.838700] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3964.696449] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3966.556429] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3977.192080] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 3986.401274] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3986.403273] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3986.406444] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3987.425905] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3987.428055] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3987.428111] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3987.433225] Lustre: Skipped 1 previous similar message [ 3991.766825] Lustre: server umount lustre-OST0000 complete [ 3996.641133] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3996.643469] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3996.648473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4002.785510] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4002.789194] Lustre: Skipped 3 previous similar messages [ 4007.392109] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4007.456968] Lustre: server umount lustre-MDT0000 complete [ 4007.905340] LustreError: 260298:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4007.909680] LustreError: 260298:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4008.741287] LustreError: 260256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772682246 with bad export cookie 4484189660272253786 [ 4008.744085] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4008.744705] LustreError: 260256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4008.862466] Lustre: server umount lustre-MDT0001 complete [ 4012.649142] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 4013.597112] Key type lgssc unregistered [ 4013.724442] LNet: 263660:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4013.727068] LNetError: 263660:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4013.732287] LNet: Removed LNI 192.168.201.103@tcp [ 4014.078907] Key type .llcrypt unregistered [ 4014.080028] Key type ._llcrypt unregistered [ 4017.983559] Key type ._llcrypt registered [ 4017.985251] Key type .llcrypt registered [ 4018.022446] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 4023.821572] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 4029.928177] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 4030.217951] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4030.235932] alg: No test for adler32 (adler32-zlib) [ 4031.094145] Lustre: Lustre: Build Version: 2.17.50_193_g407ab0b [ 4031.175865] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 4032.752163] Key type lgssc registered [ 4033.112211] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4045.565737] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4045.570600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4046.669094] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4046.679802] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4046.726916] Lustre: lustre-MDT0000: new disk, initializing [ 4046.751247] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4046.757338] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4048.117100] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4052.779125] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4052.848463] Lustre: 267130:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4052.889259] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4052.892901] Lustre: Skipped 1 previous similar message [ 4052.941746] Lustre: lustre-MDT0001: new disk, initializing [ 4052.984557] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4052.999704] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4053.005652] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4054.664205] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4058.259722] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4059.602453] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4062.016507] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4062.122819] Lustre: lustre-OST0000: new disk, initializing [ 4062.124777] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4062.151240] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4062.211396] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4062.217301] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4062.265918] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4064.285993] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4067.387742] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4068.708112] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4068.782320] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4070.221700] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4070.318505] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4072.422727] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4072.423778] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4072.424815] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4072.430808] Lustre: Skipped 1 previous similar message [ 4072.435571] LustreError: Skipped 1 previous similar message [ 4077.538198] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4077.540369] Lustre: Skipped 2 previous similar messages [ 4082.660604] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4082.662957] Lustre: Skipped 1 previous similar message [ 4085.216121] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4085.274427] Lustre: server umount lustre-OST0000 complete [ 4087.778129] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4087.778739] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4087.781465] Lustre: Skipped 1 previous similar message [ 4087.784419] Lustre: Skipped 1 previous similar message [ 4092.793871] Lustre: server umount lustre-MDT0000 complete [ 4094.354932] LustreError: 267125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772682332 with bad export cookie 15635481357759904137 [ 4094.359533] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4094.363959] LustreError: 267125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4094.500352] Lustre: server umount lustre-MDT0001 complete [ 4097.968299] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 22:45:36 (1772682336) [ 4100.430946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4100.617189] LustreError: 271135:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4100.623735] LustreError: 271135:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4100.645168] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4102.223289] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4105.478090] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4105.609611] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4107.317671] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4109.069054] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4110.470680] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4113.148658] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4115.235678] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4117.190723] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4130.788771] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4130.789685] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4130.790710] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4130.802073] Lustre: Skipped 1 previous similar message [ 4136.746624] Lustre: server umount lustre-OST0000 complete [ 4141.028675] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4141.031451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4141.036068] Lustre: Skipped 1 previous similar message [ 4141.040075] Lustre: Skipped 3 previous similar messages [ 4144.480499] Lustre: server umount lustre-MDT0000 complete [ 4145.779881] LustreError: 271115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772682383 with bad export cookie 15635481357759904949 [ 4145.785174] LustreError: 271854:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000bd2:0x1:0x0]: rc = -5 [ 4145.786593] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4145.790713] LustreError: 271854:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000b91dd4bc x1858792082955904/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 264/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4145.937414] Lustre: server umount lustre-MDT0001 complete [ 4149.976532] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 4151.091657] Key type lgssc unregistered [ 4151.221339] LNet: 274335:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4151.223670] LNetError: 274335:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4151.232276] LNet: Removed LNI 192.168.201.103@tcp [ 4151.530541] Key type .llcrypt unregistered [ 4151.531707] Key type ._llcrypt unregistered [ 4161.747709] Key type ._llcrypt registered [ 4161.749285] Key type .llcrypt registered [ 4161.786742] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 4168.543521] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 4168.830770] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4168.837842] alg: No test for adler32 (adler32-zlib) [ 4169.683561] Lustre: Lustre: Build Version: 2.17.50_193_g407ab0b [ 4169.758153] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 4171.344274] Key type lgssc registered [ 4171.834325] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4187.529854] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4187.538074] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4188.653035] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4188.665522] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4188.703926] Lustre: lustre-MDT0000: new disk, initializing [ 4188.726499] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4188.732185] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4189.901429] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4194.513849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4194.549605] Lustre: 278096:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4194.569640] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4194.571194] Lustre: Skipped 1 previous similar message [ 4194.606711] Lustre: lustre-MDT0001: new disk, initializing [ 4194.634945] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4194.645289] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4194.647871] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4195.777689] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4199.221110] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4200.281286] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4202.839239] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4202.982113] Lustre: lustre-OST2710: new disk, initializing [ 4202.985457] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 4203.022097] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 4204.162880] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 4204.170242] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 4204.212305] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 4205.028501] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4208.057633] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 4210.603889] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4210.673374] Lustre: lustre-OST03e8: new disk, initializing [ 4210.675563] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 4210.712495] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 4211.354369] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 4211.360928] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 4211.400315] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 4213.333426] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4216.074415] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 4220.367409] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 4220.471739] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4222.164235] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 4222.265489] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4224.093832] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 4224.186310] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4226.107271] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 4226.209914] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4230.625404] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4230.627694] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4230.632355] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4232.162585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4232.166630] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4232.176537] Lustre: Skipped 1 previous similar message [ 4233.929395] Lustre: server umount lustre-MDT0000 complete [ 4235.746549] LustreError: 278089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772682473 with bad export cookie 15241626354956171238 [ 4235.749781] LustreError: 278089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4235.751458] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4235.895024] Lustre: server umount lustre-MDT0001 complete [ 4248.103344] Lustre: server umount lustre-OST2710 complete [ 4259.232833] Lustre: server umount lustre-OST03e8 complete [ 4266.008825] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 4268.904207] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 4280.237409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4280.322286] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4280.334887] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4280.369168] Lustre: lustre-MDT0000: new disk, initializing [ 4280.390185] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4280.396265] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4281.715776] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4286.059063] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4286.106303] Lustre: 285814:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4286.212253] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4287.649474] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4290.839110] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4292.124923] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4294.478316] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4294.582963] Lustre: lustre-OST0000: new disk, initializing [ 4294.584832] Lustre: Skipped 1 previous similar message [ 4294.587110] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4294.589963] Lustre: Skipped 2 previous similar messages [ 4294.622732] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4294.626100] Lustre: Skipped 1 previous similar message [ 4295.799359] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4295.801990] Lustre: Skipped 1 previous similar message [ 4295.804479] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4295.838389] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4296.672368] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4299.793704] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4301.183047] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4301.265175] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4302.629432] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4302.697512] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4305.433984] Lustre: server umount lustre-OST0000 complete [ 4306.912882] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4306.915513] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4306.918644] Lustre: Skipped 1 previous similar message [ 4306.920487] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4306.922135] Lustre: Skipped 3 previous similar messages [ 4311.521125] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4311.521729] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4316.641723] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4316.643636] Lustre: Skipped 2 previous similar messages [ 4316.668402] Lustre: server umount lustre-MDT0000 complete [ 4317.880144] LustreError: 285806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772682555 with bad export cookie 15241626354956172911 [ 4317.883370] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4317.883842] LustreError: 285806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4327.454143] Lustre: server umount lustre-MDT0001 complete [ 4330.645456] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 22:49:28 (1772682568) [ 4331.236498] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 4331.854930] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 22:49:30 (1772682570) [ 4332.926576] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 4341.687756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4341.805481] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4341.813140] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4341.875601] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4343.336675] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4346.689343] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4346.735260] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4346.749038] Lustre: 291243:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4348.332916] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4350.356207] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4351.588423] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4364.924331] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4364.996123] Lustre: Denying initial registration attempt from nid 192.168.201.103@tcp, specified as failover [ 4364.999482] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 4365.003135] LustreError: 292854:0:(tgt_mount.c:2579:server_fill_super()) Unable to start targets: -99 [ 4365.005180] LustreError: 292854:0:(tgt_mount.c:2066:server_put_super()) no obd lustre-OST0000 [ 4365.007354] LustreError: 292854:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 4365.033436] Lustre: server umount lustre-OST0000 complete [ 4365.035371] LustreError: 292854:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 4367.328843] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4367.331548] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4367.336137] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4373.392721] Lustre: server umount lustre-MDT0000 complete [ 4374.602575] LustreError: 293224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772682612 with bad export cookie 15241626354956173779 [ 4374.606298] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4374.606871] LustreError: 293224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4374.712116] Lustre: server umount lustre-MDT0001 complete [ 4378.028373] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 4379.054642] Key type lgssc unregistered [ 4379.182377] LNet: 293958:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4379.185983] LNetError: 293958:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4379.197827] LNet: Removed LNI 192.168.201.103@tcp [ 4379.512519] Key type .llcrypt unregistered [ 4379.513896] Key type ._llcrypt unregistered [ 4383.042197] Key type ._llcrypt registered [ 4383.043152] Key type .llcrypt registered [ 4383.080896] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 4383.406101] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4383.429602] alg: No test for adler32 (adler32-zlib) [ 4384.294179] Lustre: Lustre: Build Version: 2.17.50_193_g407ab0b [ 4384.388623] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 4385.976173] Key type lgssc registered [ 4386.328699] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4420.982802] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 4423.777418] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4423.783459] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4424.886627] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4424.895702] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4424.955267] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4426.211690] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4429.070575] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4429.109587] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4429.121510] Lustre: 296679:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4429.216155] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4430.598699] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4432.332361] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4433.522775] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4446.890763] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4446.962773] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4447.020769] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4448.809010] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4450.565535] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4451.275013] Lustre: server umount lustre-OST0000 complete [ 4454.880941] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4454.885050] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4454.891515] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4457.953255] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4457.953805] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4458.640256] Lustre: server umount lustre-MDT0000 complete [ 4459.852343] LustreError: 296671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772682697 with bad export cookie 3033925236777529911 [ 4459.856494] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4459.858701] LustreError: 296671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4459.965312] Lustre: server umount lustre-MDT0001 complete [ 4463.396456] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 4464.367629] Key type lgssc unregistered [ 4464.502363] LNet: 299884:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4464.505268] LNetError: 299884:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4464.514301] LNet: Removed LNI 192.168.201.103@tcp [ 4464.808062] Key type .llcrypt unregistered [ 4464.809163] Key type ._llcrypt unregistered [ 4473.877085] Key type ._llcrypt registered [ 4473.878026] Key type .llcrypt registered [ 4473.914285] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 4474.376650] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4474.387243] alg: No test for adler32 (adler32-zlib) [ 4475.250054] Lustre: Lustre: Build Version: 2.17.50_193_g407ab0b [ 4475.343203] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 4476.928540] Key type lgssc registered [ 4477.363464] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4480.490926] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4480.499724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4481.730047] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4483.320706] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4486.779129] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4486.919518] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4488.486208] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4490.613879] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4491.924216] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4494.369449] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4494.486376] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4496.577684] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4498.654336] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4512.227342] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4512.228789] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4512.232674] Lustre: Skipped 1 previous similar message [ 4514.171285] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.3@tcp (stopping) [ 4514.174663] Lustre: Skipped 2 previous similar messages [ 4517.347264] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4517.349400] Lustre: Skipped 2 previous similar messages [ 4518.123703] Lustre: server umount lustre-MDT0000 complete [ 4519.282181] LustreError: 301256:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4519.292427] LustreError: 301256:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4519.835514] LustreError: 301241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772682757 with bad export cookie 14735571083386797695 [ 4519.839257] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4519.848644] LustreError: 301241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4522.465098] LustreError: 303193:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4522.465688] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4522.477769] Lustre: Skipped 1 previous similar message [ 4522.480808] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4524.403382] LustreError: 301254:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4529.523333] LustreError: 302513:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4529.523755] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.3@tcp (stopping) [ 4529.531971] LustreError: 302513:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4529.541328] Lustre: Skipped 2 previous similar messages [ 4534.240117] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4534.340569] Lustre: server umount lustre-MDT0001 complete [ 4536.839495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4540.089761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4544.994266] LustreError: 300570:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000b7ba6c84 x1858792548491008/t0(0) o250->MGC192.168.201.103@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4545.143333] LustreError: 302862:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4545.149774] LustreError: 302862:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4545.182820] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4545.210876] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [ 4546.906350] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4550.319808] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4550.482837] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4551.979855] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4553.912374] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4559.846424] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4561.641553] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 7 sec [ 4562.856315] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4563.396496] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4564.963086] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4564.964852] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4564.967847] Lustre: Skipped 1 previous similar message [ 4564.970107] Lustre: Skipped 1 previous similar message [ 4570.846160] Lustre: server umount lustre-OST0000 complete [ 4575.202139] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4578.472643] Lustre: server umount lustre-MDT0000 complete [ 4579.993502] LustreError: 301239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772682817 with bad export cookie 14735571083386804814 [ 4579.997814] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4579.999750] LustreError: 301239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4580.133337] Lustre: server umount lustre-MDT0001 complete [ 4584.228030] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 4585.545963] Key type lgssc unregistered [ 4585.699365] LNet: 306964:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4585.702387] LNetError: 306964:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4585.715638] LNet: Removed LNI 192.168.201.103@tcp [ 4586.066580] Key type .llcrypt unregistered [ 4586.068638] Key type ._llcrypt unregistered [ 4595.972946] Key type ._llcrypt registered [ 4595.973894] Key type .llcrypt registered [ 4596.018533] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 4596.464397] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4596.517075] alg: No test for adler32 (adler32-zlib) [ 4597.407890] Lustre: Lustre: Build Version: 2.17.50_193_g407ab0b [ 4597.515916] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 4599.120170] Key type lgssc registered [ 4599.658535] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4602.873514] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4602.879344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4604.076605] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4605.641651] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4608.810807] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4608.958904] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4610.466671] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4612.530669] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4614.009814] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4618.273712] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4618.274535] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4618.277073] Lustre: Skipped 1 previous similar message [ 4621.345084] Lustre: server umount lustre-MDT0000 complete [ 4622.744715] LustreError: 308322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772682860 with bad export cookie 11416356764591792827 [ 4622.750227] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4622.750798] LustreError: 308322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4622.886850] Lustre: server umount lustre-MDT0001 complete [ 4628.171615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4628.287312] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4628.298880] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4628.387403] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4629.868320] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4633.243149] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4633.290203] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4633.305399] Lustre: 310702:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4633.416616] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4634.981838] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4637.176233] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4638.660406] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4641.879625] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4641.987310] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 4641.993584] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 4641.998449] LustreError: 312357:0:(tgt_mount.c:2579:server_fill_super()) Unable to start targets: -2 [ 4642.001771] LustreError: 312357:0:(tgt_mount.c:2066:server_put_super()) no obd lustre-OST0000 [ 4642.007818] LustreError: 312357:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 4642.040733] Lustre: server umount lustre-OST0000 complete [ 4642.044296] LustreError: 312357:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 4645.431299] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4645.513746] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4645.595744] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4647.799195] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4649.900800] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4652.949440] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4652.987653] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4653.019666] Lustre: lustre-OST0001: new disk, initializing [ 4653.022428] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4654.811625] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4654.817233] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4654.844942] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [ 4654.888468] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4655.562805] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4657.939073] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4660.198972] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4660.199685] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4660.200215] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4660.200220] Lustre: Skipped 1 previous similar message [ 4660.203447] LustreError: Skipped 1 previous similar message [ 4660.209339] Lustre: Skipped 1 previous similar message [ 4661.129914] Lustre: server umount lustre-OST0001 complete [ 4665.319327] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4665.319484] LustreError: 312823:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4665.320376] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4665.321608] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4665.321612] Lustre: Skipped 1 previous similar message [ 4665.322747] LustreError: Skipped 1 previous similar message [ 4665.336995] LustreError: 312823:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4670.435922] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4670.435942] LustreError: 314027:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4670.438865] Lustre: Skipped 2 previous similar messages [ 4670.445271] LustreError: 314027:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4675.554755] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4675.557352] LustreError: 312821:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4675.559988] Lustre: Skipped 1 previous similar message [ 4675.568950] LustreError: 312821:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4677.088122] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4677.151184] Lustre: server umount lustre-OST0000 complete [ 4679.649276] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4679.653792] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4679.662052] Lustre: Skipped 1 previous similar message [ 4684.830685] Lustre: server umount lustre-MDT0000 complete [ 4685.793245] LustreError: 310713:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4685.800810] LustreError: 310713:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4686.360539] LustreError: 311423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772682924 with bad export cookie 11416356764591793366 [ 4686.364758] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4686.367745] LustreError: 311423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4686.508400] Lustre: server umount lustre-MDT0001 complete [ 4690.501025] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 4691.628356] Key type lgssc unregistered [ 4691.758551] LNet: 315607:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4691.762336] LNetError: 315607:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4691.773323] LNet: Removed LNI 192.168.201.103@tcp [ 4692.088182] Key type .llcrypt unregistered [ 4692.089188] Key type ._llcrypt unregistered [ 4727.673909] Key type ._llcrypt registered [ 4727.676925] Key type .llcrypt registered [ 4727.733589] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 4734.794663] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 4735.378743] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4735.388058] alg: No test for adler32 (adler32-zlib) [ 4736.276656] Lustre: Lustre: Build Version: 2.17.50_193_g407ab0b [ 4736.379802] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 4737.968148] Key type lgssc registered [ 4738.421412] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4752.596637] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4752.604885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4753.732409] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4753.755369] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4753.806742] Lustre: lustre-MDT0000: new disk, initializing [ 4753.847536] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4753.857796] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4755.653384] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4760.757716] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4760.821227] Lustre: 320476:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4760.866247] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4760.868864] Lustre: Skipped 1 previous similar message [ 4760.911958] Lustre: lustre-MDT0001: new disk, initializing [ 4760.957185] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4760.973559] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4760.978041] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4762.868119] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4766.675894] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4768.297515] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4770.981484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4771.100808] Lustre: lustre-OST0000: new disk, initializing [ 4771.103402] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4771.133208] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4773.345267] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4776.683380] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4778.354974] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4778.499618] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4778.503334] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4778.547084] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4779.477940] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 4780.963059] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4781.043890] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4783.585875] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4783.587609] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4783.588034] LustreError: Skipped 1 previous similar message [ 4783.592746] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4783.595730] Lustre: Skipped 1 previous similar message [ 4787.807146] Lustre: server umount lustre-OST0000 complete [ 4791.777294] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4791.779969] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4791.784573] Lustre: Skipped 1 previous similar message [ 4791.786389] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4793.825900] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4793.826113] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4795.556745] Lustre: server umount lustre-MDT0000 complete [ 4796.978406] LustreError: 322217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772683034 with bad export cookie 11969690053244190718 [ 4796.982565] LustreError: 322217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4796.982801] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4797.127463] Lustre: server umount lustre-MDT0001 complete [ 4800.423580] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 22:57:18 (1772683038) [ 4803.481963] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 22:57:21 (1772683041) [ 4806.871775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4807.070951] LustreError: 324829:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4807.078646] LustreError: 324829:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4807.099334] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4808.712373] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4812.258627] LustreError: 324830:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4812.301570] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4812.480481] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4814.245425] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4816.716299] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4818.321517] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4821.057814] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4821.195744] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4823.538704] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4825.859447] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4839.909988] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 4840.635989] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 4841.299508] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 4841.995400] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 4842.711508] Lustre: Failing over lustre-MDT0000 [ 4842.904690] Lustre: server umount lustre-MDT0000 complete [ 4843.488683] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4843.491250] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4843.495514] LustreError: 325957:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4846.459390] LustreError: 324854:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4851.570813] LustreError: 324825:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4851.576039] LustreError: 324825:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4856.299595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4856.351319] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4856.457117] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4856.488698] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4856.690084] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4857.985435] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4861.926543] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4861.940027] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 4861.959394] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 4863.351090] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4863.955393] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4865.919911] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 4866.628727] Lustre: Failing over lustre-MDT0000 [ 4866.803303] Lustre: server umount lustre-MDT0000 complete [ 4866.936071] LustreError: 324824:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4866.944609] LustreError: 324824:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4867.042363] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4867.043475] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4867.048760] Lustre: Skipped 4 previous similar messages [ 4880.061839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4880.110708] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4880.262632] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4881.855728] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4882.293829] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4885.483665] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4885.486069] Lustre: Skipped 2 previous similar messages [ 4885.500888] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 4885.519375] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 4886.924806] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4887.573711] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4890.594641] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4890.595421] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4890.598989] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4890.606115] Lustre: Skipped 1 previous similar message [ 4890.610599] Lustre: Skipped 3 previous similar messages [ 4895.714421] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4895.717829] Lustre: Skipped 2 previous similar messages [ 4896.226406] Lustre: server umount lustre-MDT0000 complete [ 4897.655628] LustreError: 324810:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772683135 with bad export cookie 11969690053244193413 [ 4897.659675] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4897.661325] LustreError: 324810:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4897.797068] Lustre: server umount lustre-MDT0001 complete [ 4909.544898] Lustre: server umount lustre-OST0000 complete [ 4915.700884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4915.883190] LustreError: 330626:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4915.891122] LustreError: 330626:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 4915.913350] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4915.921088] Lustre: Skipped 1 previous similar message [ 4917.617889] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4921.030172] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4922.859127] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4924.949387] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4926.424277] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4929.089419] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4931.377417] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4933.488875] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4940.522070] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 4945.889432] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4945.893425] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4945.895079] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4945.900401] Lustre: Skipped 2 previous similar messages [ 4945.907388] Lustre: Skipped 1 previous similar message [ 4960.224107] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4960.286547] Lustre: server umount lustre-OST0000 complete [ 4962.273070] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4962.277677] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4962.280325] Lustre: Skipped 4 previous similar messages [ 4976.096932] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4976.172639] Lustre: server umount lustre-MDT0000 complete [ 4976.609323] LustreError: 330627:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4976.617836] LustreError: 330627:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4977.802374] LustreError: 330606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772683215 with bad export cookie 11969690053244194134 [ 4977.806363] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4977.807883] LustreError: 330606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4977.969827] Lustre: server umount lustre-MDT0001 complete [ 4982.562810] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 4983.894255] Key type lgssc unregistered [ 4984.053353] LNet: 333829:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4984.056717] LNetError: 333829:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4984.066561] LNet: Removed LNI 192.168.201.103@tcp [ 4984.417870] Key type .llcrypt unregistered [ 4984.419036] Key type ._llcrypt unregistered [ 4995.223994] Key type ._llcrypt registered [ 4995.224916] Key type .llcrypt registered [ 4995.264841] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 4995.615508] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4995.634678] alg: No test for adler32 (adler32-zlib) [ 4996.494741] Lustre: Lustre: Build Version: 2.17.50_193_g407ab0b [ 4996.585138] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 4998.168359] Key type lgssc registered [ 4998.575291] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5001.370922] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5001.379048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5002.575292] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5004.083997] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5007.533138] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5007.698856] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5009.394618] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5011.610905] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5013.055427] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5015.626884] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5015.763229] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5017.897647] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5020.134048] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5028.078977] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 5038.562343] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5038.564224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5038.566851] Lustre: Skipped 2 previous similar messages [ 5038.569381] Lustre: Skipped 1 previous similar message [ 5038.573156] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5042.080824] Lustre: server umount lustre-MDT0000 complete [ 5043.533885] LustreError: 335285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772683281 with bad export cookie 15419951054510832434 [ 5043.538429] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5043.539434] LustreError: 335285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5043.665349] Lustre: server umount lustre-MDT0001 complete [ 5055.465738] Lustre: server umount lustre-OST0000 complete [ 5060.752452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5060.940192] LustreError: 338710:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5060.947220] LustreError: 338710:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5060.965121] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5062.420477] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5065.711394] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5065.879949] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5067.581958] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5069.767963] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5071.023776] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5073.395567] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5075.651087] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5077.662934] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5085.799852] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5096.417848] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5096.420721] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5096.422856] Lustre: Skipped 2 previous similar messages [ 5096.424697] Lustre: Skipped 2 previous similar messages [ 5101.149697] Lustre: server umount lustre-MDT0000 complete [ 5101.539087] LustreError: 340864:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5101.548906] LustreError: 340864:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5102.509332] LustreError: 340325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772683340 with bad export cookie 15419951054510833533 [ 5102.512713] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5102.513776] LustreError: 340325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5102.638912] Lustre: server umount lustre-MDT0001 complete [ 5113.306798] Lustre: server umount lustre-OST0000 complete [ 5119.220895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5119.390629] LustreError: 342305:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5119.416308] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5119.418544] Lustre: Skipped 1 previous similar message [ 5120.887965] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5124.378797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5125.947503] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5127.965762] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5129.336512] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5131.849564] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5134.105286] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5136.034300] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5144.300779] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5149.666900] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5149.667964] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5149.675083] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5149.677100] Lustre: Skipped 2 previous similar messages [ 5153.886278] Lustre: server umount lustre-OST0000 complete [ 5159.906272] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5159.907088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5159.911112] Lustre: Skipped 2 previous similar messages [ 5161.628727] Lustre: server umount lustre-MDT0000 complete [ 5163.006264] LustreError: 342285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772683400 with bad export cookie 15419951054510834541 [ 5163.008675] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5163.009884] LustreError: 342285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5163.137864] Lustre: server umount lustre-MDT0001 complete [ 5167.068738] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 5168.216270] Key type lgssc unregistered [ 5168.356578] LNet: 345504:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5168.360510] LNetError: 345504:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5168.372513] LNet: Removed LNI 192.168.201.103@tcp [ 5168.674953] Key type .llcrypt unregistered [ 5168.676272] Key type ._llcrypt unregistered [ 5178.774699] Key type ._llcrypt registered [ 5178.776222] Key type .llcrypt registered [ 5178.821416] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 5179.258746] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5179.304039] alg: No test for adler32 (adler32-zlib) [ 5180.176954] Lustre: Lustre: Build Version: 2.17.50_193_g407ab0b [ 5180.266119] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 5181.856155] Key type lgssc registered [ 5182.315413] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5185.426297] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5185.435361] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5185.440180] LustreError: 346902:0:(osd_handler.c:8531:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 5185.451519] LustreError: 346902:0:(obd_config.c:880:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 5185.455243] LustreError: 346902:0:(obd_mount.c:217:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 5185.459617] LustreError: 346902:0:(tgt_mount.c:2547:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 5185.464079] LustreError: 346902:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5188.390793] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5188.393871] LustreError: 347299:0:(osd_handler.c:8531:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 5188.402176] LustreError: 347299:0:(obd_config.c:880:class_setup()) setup lustre-OST0000-osd failed (-22) [ 5188.404373] LustreError: 347299:0:(obd_mount.c:217:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 5188.406823] LustreError: 347299:0:(tgt_mount.c:2547:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 5188.410110] LustreError: 347299:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5195.093348] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 5196.153557] Key type lgssc unregistered [ 5196.291351] LNet: 348283:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5196.293381] LNetError: 348283:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5196.304521] LNet: Removed LNI 192.168.201.103@tcp [ 5196.607381] Key type .llcrypt unregistered [ 5196.608946] Key type ._llcrypt unregistered [ 5203.125445] Key type ._llcrypt registered [ 5203.126611] Key type .llcrypt registered [ 5203.167347] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 5210.159224] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 5210.584140] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5210.637187] alg: No test for adler32 (adler32-zlib) [ 5211.494676] Lustre: Lustre: Build Version: 2.17.50_193_g407ab0b [ 5211.593819] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 5213.176280] Key type lgssc registered [ 5213.574697] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5226.629337] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5226.635945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5227.741232] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5227.754268] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5227.800499] Lustre: lustre-MDT0000: new disk, initializing [ 5227.826088] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5227.833417] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5229.247694] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5233.623454] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5233.670819] Lustre: 351379:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5233.701456] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5233.705238] Lustre: Skipped 1 previous similar message [ 5233.748264] Lustre: lustre-MDT0001: new disk, initializing [ 5233.781675] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5233.798640] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5233.802474] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5235.214802] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5238.608500] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5240.087390] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5242.731490] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5242.857850] Lustre: lustre-OST0000: new disk, initializing [ 5242.860953] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5242.864449] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 5242.916397] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5245.059883] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5247.977897] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5247.981628] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5247.998362] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5248.317755] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5249.794098] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5249.875648] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5251.422938] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5251.509396] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5253.092293] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5253.092397] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5253.094670] LustreError: Skipped 1 previous similar message [ 5253.095575] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5253.098616] Lustre: Skipped 1 previous similar message [ 5254.301130] Lustre: server umount lustre-OST0000 complete [ 5258.210196] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5258.210964] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5258.214048] Lustre: Skipped 1 previous similar message [ 5258.217392] Lustre: Skipped 2 previous similar messages [ 5261.869374] Lustre: server umount lustre-MDT0000 complete [ 5263.347091] LustreError: 351370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772683501 with bad export cookie 13391210398060208688 [ 5263.350462] LustreError: 351370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5263.351172] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5263.483088] Lustre: server umount lustre-MDT0001 complete [ 5266.780605] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 5267.428505] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 23:05:05 (1772683505) [ 5270.047394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5270.238552] LustreError: 355431:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5270.249706] LustreError: 355431:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5270.276789] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5271.963116] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5275.335072] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5275.493329] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5277.120580] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5279.323552] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5280.667595] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5283.081305] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5285.323582] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5287.454170] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5290.210365] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5290.282556] Lustre: lustre-OST0001: new disk, initializing [ 5290.284805] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5290.321616] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5290.324922] Lustre: Skipped 1 previous similar message [ 5292.588600] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5296.143117] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5296.148254] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5296.168170] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5296.191082] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5301.219607] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5301.219631] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5301.221896] LustreError: Skipped 1 previous similar message [ 5301.223281] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5301.225266] Lustre: Skipped 1 previous similar message [ 5301.229493] Lustre: Skipped 1 previous similar message [ 5304.129355] Lustre: server umount lustre-OST0001 complete [ 5306.339664] LustreError: 357357:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5306.344238] LustreError: 357357:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5311.462502] LustreError: 357357:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5311.469576] LustreError: 357357:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5316.579949] LustreError: 357024:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5316.587724] LustreError: 357024:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5321.185496] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5321.185528] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5321.188878] LustreError: Skipped 1 previous similar message [ 5321.189627] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5321.193329] Lustre: Skipped 1 previous similar message [ 5321.200331] Lustre: Skipped 2 previous similar messages [ 5321.701124] LustreError: 357025:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5321.707148] LustreError: 357025:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5322.331563] Lustre: server umount lustre-OST0000 complete [ 5326.818568] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5326.822114] Lustre: Skipped 1 previous similar message [ 5326.823747] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5329.821195] Lustre: server umount lustre-MDT0000 complete [ 5331.062612] LustreError: 355411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772683568 with bad export cookie 13391210398060209528 [ 5331.066361] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5331.068634] LustreError: 355411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5337.288606] Lustre: server umount lustre-MDT0001 complete [ 5341.143466] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 5342.296230] Key type lgssc unregistered [ 5342.426331] LNet: 359783:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5342.428739] LNetError: 359783:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5342.436281] LNet: Removed LNI 192.168.201.103@tcp [ 5342.747498] Key type .llcrypt unregistered [ 5342.748355] Key type ._llcrypt unregistered [ 5360.135281] Key type ._llcrypt registered [ 5360.136962] Key type .llcrypt registered [ 5360.176169] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 5360.510224] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5360.552312] alg: No test for adler32 (adler32-zlib) [ 5361.403241] Lustre: Lustre: Build Version: 2.17.50_193_g407ab0b [ 5361.484926] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 5363.072107] Key type lgssc registered [ 5363.454030] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5366.133512] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5366.140441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5367.234858] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5367.243176] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5367.308677] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5368.700326] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5371.627710] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5371.665487] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5371.676917] Lustre: 362540:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5371.783409] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5373.164592] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5374.991960] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5376.219313] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5376.992856] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5376.995436] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5376.999493] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5378.017322] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5378.017821] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5383.137175] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5383.139709] Lustre: Skipped 2 previous similar messages [ 5388.257168] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5388.258900] Lustre: Skipped 1 previous similar message [ 5391.328175] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5391.395820] Lustre: server umount lustre-MDT0000 complete [ 5392.761572] LustreError: 362529:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772683630 with bad export cookie 607030783566117745 [ 5392.764644] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5392.766793] LustreError: 362529:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5392.890956] Lustre: server umount lustre-MDT0001 complete [ 5396.173327] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 23:07:14 (1772683634) [ 5398.399971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5398.590377] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5399.848018] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5402.452396] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5402.560064] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5403.765027] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5405.429267] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5406.544427] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5408.691369] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5408.756991] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5408.820030] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5410.605302] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5412.292406] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5421.234403] Lustre: Permanently deactivating lustre-OST0000 [ 5421.239222] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 5423.662297] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5424.319766] LustreError: 367373:0:(mgs_llog.c:1942:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 5424.322197] LustreError: 367373:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [ 5424.324508] LustreError: 367373:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5424.838771] LustreError: 367421:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [ 5424.840708] LustreError: 367421:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5431.508937] Lustre: server umount lustre-OST0000 complete [ 5432.399335] LustreError: 367670:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [ 5432.401514] LustreError: 367670:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5433.312891] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5433.315078] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5433.319476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5442.552442] Lustre: server umount lustre-MDT0000 complete [ 5443.805358] LustreError: 364803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772683681 with bad export cookie 607030783566118312 [ 5443.808275] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5443.809679] LustreError: 364803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5453.327812] Lustre: server umount lustre-MDT0001 complete [ 5456.165682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5457.443242] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5458.352393] LustreError: 368809:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 5460.372208] Lustre: 369148:0:(mgs_llog.c:1709:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.103@tcp [ 5461.022196] Lustre: server umount MGS complete [ 5464.729822] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5464.880216] LustreError: 369871:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5464.885037] LustreError: 369871:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5464.885630] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5464.890459] Lustre: Skipped 1 previous similar message [ 5464.911350] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5466.140258] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5468.936855] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5469.045856] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5470.446781] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5472.342838] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5473.548027] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5474.211848] Lustre: Permanently reactivating lustre-OST0000 [ 5474.217531] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 5474.219446] Lustre: Skipped 2 previous similar messages [ 5499.360118] LustreError: 371423:0:(osp_dev.c:694:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 5502.372314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5502.486482] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5502.488565] Lustre: Skipped 1 previous similar message [ 5504.318237] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5506.062224] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5518.816728] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5518.818819] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5518.821974] Lustre: Skipped 1 previous similar message [ 5518.823371] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5518.824906] Lustre: Skipped 4 previous similar messages [ 5524.049485] Lustre: server umount lustre-OST0000 complete [ 5525.473050] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5525.475131] LustreError: Skipped 1 previous similar message [ 5525.476283] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5525.479388] Lustre: Skipped 1 previous similar message [ 5531.482266] Lustre: server umount lustre-MDT0000 complete [ 5532.725984] LustreError: 369851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772683770 with bad export cookie 607030783566119348 [ 5532.729116] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5532.729670] LustreError: 369851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5532.841473] Lustre: server umount lustre-MDT0001 complete [ 5536.572376] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 5537.675991] Key type lgssc unregistered [ 5537.804508] LNet: 374186:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5537.806654] LNetError: 374186:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5537.816431] LNet: Removed LNI 192.168.201.103@tcp [ 5538.161931] Key type .llcrypt unregistered [ 5538.163273] Key type ._llcrypt unregistered [ 5544.202825] Key type ._llcrypt registered [ 5544.203929] Key type .llcrypt registered [ 5544.242283] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 5550.704705] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 5551.093213] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5551.136934] alg: No test for adler32 (adler32-zlib) [ 5551.991636] Lustre: Lustre: Build Version: 2.17.50_193_g407ab0b [ 5552.079738] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 5553.664115] Key type lgssc registered [ 5554.039428] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5565.312321] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5565.317548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5566.407721] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5566.418478] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5566.456123] Lustre: lustre-MDT0000: new disk, initializing [ 5566.481933] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5566.488608] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5567.822360] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5572.068377] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5572.113399] Lustre: 377279:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5572.139607] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5572.141376] Lustre: Skipped 1 previous similar message [ 5572.182447] Lustre: lustre-MDT0001: new disk, initializing [ 5572.216334] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5572.228018] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5572.230941] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5573.585805] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5576.682092] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5577.894461] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5580.186460] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5580.290659] Lustre: lustre-OST0000: new disk, initializing [ 5580.292770] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5580.322142] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5582.206908] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5583.997052] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5584.000717] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5584.044411] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5585.256270] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5586.599334] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5586.667378] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5587.962558] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5588.038703] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5589.474846] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5589.476629] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5589.477641] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5589.480752] Lustre: Skipped 1 previous similar message [ 5589.485499] Lustre: Skipped 1 previous similar message [ 5590.734967] Lustre: server umount lustre-OST0000 complete [ 5593.056757] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5593.059376] LustreError: Skipped 1 previous similar message [ 5593.060585] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5593.063911] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5594.593034] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5594.593901] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5599.713063] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5599.714755] Lustre: Skipped 2 previous similar messages [ 5604.833651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5604.835640] Lustre: Skipped 1 previous similar message [ 5606.368123] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5606.427754] Lustre: server umount lustre-MDT0000 complete [ 5607.671423] LustreError: 377272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772683845 with bad export cookie 8239375863839623473 [ 5607.673977] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5607.675791] LustreError: 377272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5607.785169] Lustre: server umount lustre-MDT0001 complete [ 5610.543857] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 5611.092534] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 23:10:49 (1772683849) [ 5613.449834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5613.627460] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5614.903363] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5617.642223] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5617.767459] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5619.109088] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5620.908767] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5622.155950] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5624.477572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5626.615273] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5628.458613] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5629.119280] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 5640.123566] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 5643.233296] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5643.235714] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5643.239550] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5645.283851] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5647.691764] Lustre: server umount lustre-OST0000 complete [ 5650.401713] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5650.405204] Lustre: Skipped 1 previous similar message [ 5660.641136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5660.643227] Lustre: Skipped 7 previous similar messages [ 5663.200119] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5663.254946] Lustre: server umount lustre-MDT0000 complete [ 5664.454156] LustreError: 382945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772683902 with bad export cookie 8239375863839624341 [ 5664.456620] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5664.459773] LustreError: 382945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5664.565922] Lustre: server umount lustre-MDT0001 complete [ 5668.074105] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 5669.070950] Key type lgssc unregistered [ 5669.197451] LNet: 384726:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5669.199825] LNetError: 384726:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5669.208386] LNet: Removed LNI 192.168.201.103@tcp [ 5669.539888] Key type .llcrypt unregistered [ 5669.540943] Key type ._llcrypt unregistered [ 5677.263948] Key type ._llcrypt registered [ 5677.265524] Key type .llcrypt registered [ 5677.306306] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 5677.417247] Key type .llcrypt unregistered [ 5677.418590] Key type ._llcrypt unregistered [ 5684.116674] Key type ._llcrypt registered [ 5684.117507] Key type .llcrypt registered [ 5684.150626] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 5684.564574] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5684.571102] alg: No test for adler32 (adler32-zlib) [ 5685.420611] Lustre: Lustre: Build Version: 2.17.50_193_g407ab0b [ 5685.500930] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 5687.072118] Key type lgssc registered [ 5687.381625] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5689.763445] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5689.767698] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5690.907118] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5692.061540] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5694.599862] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5694.698416] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5696.464173] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5698.192213] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5700.065880] LustreError: 386879:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5700.069837] LustreError: 386879:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 5700.459948] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5700.587859] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5701.613058] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5701.961342] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5705.697834] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5705.701782] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5705.709160] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5706.723295] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5706.723913] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5706.724940] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5706.732940] Lustre: Skipped 1 previous similar message [ 5707.034163] Lustre: server umount lustre-OST0000 complete [ 5709.794960] LustreError: 388842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772683947 with bad export cookie 3597171181035500558 [ 5709.795856] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 5709.799488] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5709.808118] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5710.684327] Lustre: server umount lustre-MDT0000 complete [ 5711.841134] LustreError: 386879:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5711.841636] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5712.005297] LustreError: 389131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772683949 with bad export cookie 3597171181035500068 [ 5712.007898] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5712.009328] LustreError: 389131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5712.145779] Lustre: server umount lustre-MDT0001 complete [ 5715.802135] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 5716.799308] Key type lgssc unregistered [ 5716.936345] LNet: 389863:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5716.938990] LNetError: 389863:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5716.945284] LNet: Removed LNI 192.168.201.103@tcp [ 5717.217602] Key type .llcrypt unregistered [ 5717.218613] Key type ._llcrypt unregistered [ 5725.440665] Key type ._llcrypt registered [ 5725.441862] Key type .llcrypt registered [ 5725.478063] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 5725.808376] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5725.843621] alg: No test for adler32 (adler32-zlib) [ 5726.687374] Lustre: Lustre: Build Version: 2.17.50_193_g407ab0b [ 5726.771293] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 5728.352193] Key type lgssc registered [ 5728.665314] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5730.960300] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5730.966182] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5746.400322] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5751.520193] LustreError: 391213:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.103@tcp: failed processing log, type 4: rc = -110 [ 5778.144123] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5779.751415] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5781.280864] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5783.319195] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5783.489929] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5783.512353] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 5784.637870] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5787.176896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5787.288356] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5788.462349] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5790.116786] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5791.196806] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5797.858765] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5797.859195] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5797.860414] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5797.863243] Lustre: Skipped 1 previous similar message [ 5802.978926] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5802.981638] Lustre: Skipped 2 previous similar messages [ 5803.856257] Lustre: server umount lustre-OST0000 complete [ 5808.097696] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5808.098204] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5808.103824] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5808.105330] Lustre: Skipped 1 previous similar message [ 5811.219559] Lustre: server umount lustre-MDT0000 complete [ 5812.506068] LustreError: 391220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772684050 with bad export cookie 648026176227139646 [ 5812.510023] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5812.510798] LustreError: 391220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5812.624420] Lustre: server umount lustre-MDT0001 complete [ 5816.211837] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 5817.247945] Key type lgssc unregistered [ 5817.386458] LNet: 394427:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5817.388427] LNetError: 394427:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5817.401388] LNet: Removed LNI 192.168.201.103@tcp [ 5817.706156] Key type .llcrypt unregistered [ 5817.707181] Key type ._llcrypt unregistered [ 5826.590197] Key type ._llcrypt registered [ 5826.591167] Key type .llcrypt registered [ 5826.626530] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 5827.046539] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5827.063495] alg: No test for adler32 (adler32-zlib) [ 5827.905152] Lustre: Lustre: Build Version: 2.17.50_193_g407ab0b [ 5827.979044] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 5829.552110] Key type lgssc registered [ 5829.912710] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5832.363761] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5832.369075] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5833.530985] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5834.881918] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5837.746728] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5837.882295] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5839.140335] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5840.909913] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5842.116312] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5844.324049] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5844.447130] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5846.429400] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5848.249787] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5858.790729] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 5868.512941] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5868.516432] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5868.521559] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5869.025388] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5869.028199] Lustre: Skipped 1 previous similar message [ 5869.028284] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5870.874355] Lustre: server umount lustre-MDT0000 complete [ 5871.647400] LustreError: 397417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.3@tcp arrived at 1772684109 with bad export cookie 15640821558418460145 [ 5871.652754] LustreError: 397417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5872.309784] LustreError: 395786:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772684110 with bad export cookie 15640821558418459494 [ 5872.312825] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5872.314746] LustreError: 395786:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5873.120821] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5873.123067] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5873.126253] Lustre: Skipped 1 previous similar message [ 5873.127771] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5873.129214] Lustre: Skipped 2 previous similar messages [ 5874.144970] LustreError: 395804:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5874.155024] LustreError: 395804:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5877.152855] LustreError: 396512:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5877.153242] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5877.158262] LustreError: 396512:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5877.160337] Lustre: Skipped 1 previous similar message [ 5878.417904] Lustre: server umount lustre-OST0000 complete [ 5880.736767] Lustre: server umount lustre-MDT0001 complete [ 5884.020674] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 5884.960486] Key type lgssc unregistered [ 5885.092424] LNet: 399152:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5885.094557] LNetError: 399152:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5885.103402] LNet: Removed LNI 192.168.201.103@tcp [ 5885.388757] Key type .llcrypt unregistered [ 5885.389686] Key type ._llcrypt unregistered [ 5893.599392] Key type ._llcrypt registered [ 5893.600217] Key type .llcrypt registered [ 5893.629963] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 5893.910531] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5893.944046] alg: No test for adler32 (adler32-zlib) [ 5894.791874] Lustre: Lustre: Build Version: 2.17.50_193_g407ab0b [ 5894.865867] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 5896.440106] Key type lgssc registered [ 5896.745964] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5899.216912] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5899.223307] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5900.398652] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5901.704535] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5904.631461] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5904.757423] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5906.077148] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5907.870108] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5909.119693] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5911.421061] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5911.533461] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5913.427074] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5915.258625] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5925.866846] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 5930.464843] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 5930.468340] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5930.474535] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5930.978850] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5930.984407] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5933.845317] Lustre: server umount lustre-MDT0001 complete [ 5936.097215] LustreError: 401889:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5936.097908] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5936.098857] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5936.099405] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5936.099409] Lustre: Skipped 1 previous similar message [ 5936.103287] LustreError: 401889:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5941.217110] LustreError: 401251:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5941.217534] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5941.648386] Lustre: server umount lustre-OST0000 complete [ 5943.036105] Lustre: server umount lustre-MDT0000 complete [ 5947.499304] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 5948.610310] Key type lgssc unregistered [ 5948.751595] LNet: 403885:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5948.754052] LNetError: 403885:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5948.761454] LNet: Removed LNI 192.168.201.103@tcp [ 5949.101499] Key type .llcrypt unregistered [ 5949.103144] Key type ._llcrypt unregistered [ 5957.445660] Key type ._llcrypt registered [ 5957.446646] Key type .llcrypt registered [ 5957.484026] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 5963.629644] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 5963.986224] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5964.019801] alg: No test for adler32 (adler32-zlib) [ 5964.883901] Lustre: Lustre: Build Version: 2.17.50_193_g407ab0b [ 5964.972358] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 5966.560201] Key type lgssc registered [ 5966.943852] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5979.093668] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 5984.236815] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5984.242052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5985.337739] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5985.351177] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5985.391990] Lustre: lustre-MDT0000: new disk, initializing [ 5985.417310] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5985.423357] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5986.749969] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5992.488934] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5992.597870] Lustre: lustre-OST0000: new disk, initializing [ 5992.599640] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5992.601187] Lustre: Skipped 1 previous similar message [ 5992.627374] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5994.611168] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5997.691949] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5999.104351] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 5999.107204] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 5999.117503] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 6008.525936] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6008.579616] Lustre: 407820:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6008.617211] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6008.662439] Lustre: lustre-MDT0001: new disk, initializing [ 6008.708328] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6008.725152] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 6008.728552] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 6010.079372] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6013.681084] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6013.754973] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6015.143920] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6015.232532] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6019.554258] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6019.554962] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6019.557572] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6019.561311] Lustre: Skipped 1 previous similar message [ 6019.564372] Lustre: Skipped 1 previous similar message [ 6020.983929] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.3@tcp (stopping) [ 6024.675669] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6024.677815] Lustre: Skipped 1 previous similar message [ 6029.796478] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6029.799160] Lustre: Skipped 2 previous similar messages [ 6031.328075] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6031.383272] Lustre: server umount lustre-MDT0001 complete [ 6040.037058] LustreError: 409416:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6040.041399] LustreError: 409416:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6045.155451] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6045.155596] LustreError: 407831:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6045.156094] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6045.157069] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6045.157074] Lustre: Skipped 1 previous similar message [ 6049.876201] Lustre: server umount lustre-OST0000 complete [ 6050.273946] LustreError: 407830:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6057.463785] Lustre: server umount lustre-MDT0000 complete [ 6061.845326] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 6062.841025] Key type lgssc unregistered [ 6062.965347] LNet: 412387:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6062.968667] LNetError: 412387:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6062.980327] LNet: Removed LNI 192.168.201.103@tcp [ 6063.276088] Key type .llcrypt unregistered [ 6063.277674] Key type ._llcrypt unregistered [ 6075.923223] Key type ._llcrypt registered [ 6075.924191] Key type .llcrypt registered [ 6075.970557] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 6076.075247] Key type .llcrypt unregistered [ 6076.076836] Key type ._llcrypt unregistered [ 6082.878811] Key type ._llcrypt registered [ 6082.879796] Key type .llcrypt registered [ 6082.917500] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 6083.214870] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6083.240512] alg: No test for adler32 (adler32-zlib) [ 6084.091344] Lustre: Lustre: Build Version: 2.17.50_193_g407ab0b [ 6084.176250] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 6085.760215] Key type lgssc registered [ 6086.187641] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6094.311452] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6094.317167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6095.424289] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6095.438224] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6095.484764] Lustre: lustre-MDT0000: new disk, initializing [ 6095.510414] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6095.518091] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6096.731097] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6100.730193] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6100.788521] Lustre: 415701:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6100.809227] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6100.811508] Lustre: Skipped 1 previous similar message [ 6100.845349] Lustre: lustre-MDT0001: new disk, initializing [ 6100.872438] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6100.886098] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6100.889895] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6101.972379] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6104.886134] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6106.063447] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6108.309893] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6108.414114] Lustre: lustre-OST0000: new disk, initializing [ 6108.416895] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6108.445857] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6110.076329] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6110.079799] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6110.113501] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6110.353638] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6113.431713] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6116.320887] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6116.324188] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6116.329832] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6120.418059] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6120.419079] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6120.419847] Lustre: Skipped 1 previous similar message [ 6120.425371] Lustre: Skipped 1 previous similar message [ 6121.881293] Lustre: server umount lustre-MDT0000 complete [ 6123.126215] LustreError: 415691:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772684360 with bad export cookie 1423093349902364968 [ 6123.128876] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6123.130064] LustreError: 415691:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6123.248503] Lustre: server umount lustre-MDT0001 complete [ 6134.754634] Lustre: server umount lustre-OST0000 complete [ 6138.207661] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 23:19:36 (1772684376) [ 6140.769591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6140.945817] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6142.272684] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6145.120991] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6145.251439] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6146.621158] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6148.445885] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6149.670027] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6162.934366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6163.008704] Lustre: Found index 0 for lustre-OST0000, updating log [ 6163.074214] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6165.093634] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6166.183607] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 6166.890423] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6171.170263] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6171.171066] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6171.171929] Lustre: Skipped 1 previous similar message [ 6174.159261] Lustre: server umount lustre-OST0000 complete [ 6186.731161] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6186.736249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6186.739327] Lustre: Skipped 2 previous similar messages [ 6190.817358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6190.819329] Lustre: Skipped 3 previous similar messages [ 6201.312081] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6201.313206] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6201.316173] Lustre: Skipped 3 previous similar messages [ 6201.365867] Lustre: server umount lustre-MDT0000 complete [ 6202.473109] LustreError: 420966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772684440 with bad export cookie 1423093349902366347 [ 6202.475661] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6202.476178] LustreError: 420966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6202.584989] Lustre: server umount lustre-MDT0001 complete [ 6205.020488] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 23:20:43 (1772684443) [ 6233.696207] LNet: 1 local NIs in recovery (showing 1): 192.168.201.103@tcp [ 6784.482489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6784.631215] LustreError: 425414:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6784.637746] LustreError: 425414:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6784.655176] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6785.903510] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6788.638078] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6790.005786] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6791.719297] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6792.936908] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6806.205620] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6806.306906] Lustre: Found index 0 for lustre-OST0000, updating log [ 6808.281029] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6810.065136] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6810.744294] Lustre: server umount lustre-OST0000 complete [ 6814.688915] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6814.690844] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6814.693783] Lustre: Skipped 1 previous similar message [ 6814.695249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6815.714281] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6818.140705] Lustre: server umount lustre-MDT0000 complete [ 6819.381274] LustreError: 425394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772685057 with bad export cookie 1423093349902367404 [ 6819.384433] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6819.386448] LustreError: 425394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6819.502453] Lustre: server umount lustre-MDT0001 complete [ 6836.602079] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 6839.681916] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 6847.072227] LNet: 1 local NIs in recovery (showing 1): 192.168.201.103@tcp [ 6851.121468] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 23:31:29 (1772685089) [ 6856.758906] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 6857.776589] Key type lgssc unregistered [ 6869.152969] LNet: 431673:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6869.155967] LNetError: 431673:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6935.718345] LNet: Removed LNI 192.168.201.103@tcp [ 6936.036855] Key type .llcrypt unregistered [ 6936.038062] Key type ._llcrypt unregistered [ 6942.571751] Key type ._llcrypt registered [ 6942.573302] Key type .llcrypt registered [ 6942.613580] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 6942.940139] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6942.945981] alg: No test for adler32 (adler32-zlib) [ 6943.797806] Lustre: Lustre: Build Version: 2.17.50_193_g407ab0b [ 6943.879947] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 6945.464179] Key type lgssc registered [ 6945.836299] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6947.253933] LNet: Added LNI 192.168.201.103@tcp100 [8/256/0/180] [ 6947.550280] LNet: Added LNI 192.168.201.103@tcp101 [8/256/0/180] [ 6947.837915] LNet: Added LNI 192.168.201.103@tcp102 [8/256/0/180] [ 6948.126632] LNet: Added LNI 192.168.201.103@tcp103 [8/256/0/180] [ 6948.419192] LNet: Added LNI 192.168.201.103@tcp104 [8/256/0/180] [ 6948.718484] LNet: Added LNI 192.168.201.103@tcp105 [8/256/0/180] [ 6949.033052] LNet: Added LNI 192.168.201.103@tcp106 [8/256/0/180] [ 6949.342387] LNet: Added LNI 192.168.201.103@tcp107 [8/256/0/180] [ 6949.651646] LNet: Added LNI 192.168.201.103@tcp108 [8/256/0/180] [ 6949.976967] LNet: Added LNI 192.168.201.103@tcp109 [8/256/0/180] [ 6950.274297] LNet: Added LNI 192.168.201.103@tcp110 [8/256/0/180] [ 6950.570784] LNet: Added LNI 192.168.201.103@tcp111 [8/256/0/180] [ 6950.865082] LNet: Added LNI 192.168.201.103@tcp112 [8/256/0/180] [ 6951.162835] LNet: Added LNI 192.168.201.103@tcp113 [8/256/0/180] [ 6951.454436] LNet: Added LNI 192.168.201.103@tcp114 [8/256/0/180] [ 6951.746716] LNet: Added LNI 192.168.201.103@tcp115 [8/256/0/180] [ 6952.042836] LNet: Added LNI 192.168.201.103@tcp116 [8/256/0/180] [ 6952.341522] LNet: Added LNI 192.168.201.103@tcp117 [8/256/0/180] [ 6952.642114] LNet: Added LNI 192.168.201.103@tcp118 [8/256/0/180] [ 6952.928046] LNet: Added LNI 192.168.201.103@tcp119 [8/256/0/180] [ 6953.207848] LNet: Added LNI 192.168.201.103@tcp120 [8/256/0/180] [ 6953.486984] LNet: Added LNI 192.168.201.103@tcp121 [8/256/0/180] [ 6953.771284] LNet: Added LNI 192.168.201.103@tcp122 [8/256/0/180] [ 6954.060333] LNet: Added LNI 192.168.201.103@tcp123 [8/256/0/180] [ 6954.363823] LNet: Added LNI 192.168.201.103@tcp124 [8/256/0/180] [ 6954.653290] LNet: Added LNI 192.168.201.103@tcp125 [8/256/0/180] [ 6954.944179] LNet: Added LNI 192.168.201.103@tcp126 [8/256/0/180] [ 6955.228750] LNet: Added LNI 192.168.201.103@tcp127 [8/256/0/180] [ 6955.521728] LNet: Added LNI 192.168.201.103@tcp128 [8/256/0/180] [ 6955.806293] LNet: Added LNI 192.168.201.103@tcp129 [8/256/0/180] [ 6956.091026] LNet: Added LNI 192.168.201.103@tcp130 [8/256/0/180] [ 6956.397722] LNet: Added LNI 192.168.201.103@tcp131 [8/256/0/180] [ 6956.679144] LNet: Added LNI 192.168.201.103@tcp132 [8/256/0/180] [ 6956.967727] LNet: Added LNI 192.168.201.103@tcp133 [8/256/0/180] [ 6957.242909] LNet: Added LNI 192.168.201.103@tcp134 [8/256/0/180] [ 6957.519624] LNet: Added LNI 192.168.201.103@tcp135 [8/256/0/180] [ 6959.643420] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6959.648684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6960.693576] Lustre: 434655:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.103@tcp: can't add NID 192.168.201.103@tcp130: rc = -75 [ 6960.723237] LNetError: 432119:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.103 [ 6960.725936] LNetError: Protocol error connecting to 192.168.201.103@tcp122 at host 192.168.201.103:988: is it running a compatible version of Lustre? [ 6960.728389] LNetError: Skipped 1 previous similar message [ 6960.771199] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6960.782270] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6960.823257] Lustre: lustre-MDT0000: new disk, initializing [ 6960.854289] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6960.862124] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6961.762758] LNetError: 434676:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.103 [ 6961.765513] LNetError: 434676:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Skipped 1 previous similar message [ 6961.768306] LNetError: Protocol error connecting to 192.168.201.103@tcp104 at host 192.168.201.103:988: is it running a compatible version of Lustre? [ 6961.771933] LNetError: Skipped 3 previous similar messages [ 6962.079019] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6966.024053] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6966.068500] Lustre: 434673:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6966.091463] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6966.092987] Lustre: Skipped 1 previous similar message [ 6966.126824] Lustre: lustre-MDT0001: new disk, initializing [ 6966.137206] Lustre: 435458:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.103@tcp: can't add NID 192.168.201.103@tcp130: rc = -75 [ 6966.139592] Lustre: 435458:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 6966.180631] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6966.191674] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6966.194410] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6967.392921] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6968.503206] Lustre: 435805:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.103@tcp: can't add NID 192.168.201.103@tcp130: rc = -75 [ 6968.507772] Lustre: 435805:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 6970.333520] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6971.480077] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6974.204470] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6974.337472] Lustre: lustre-OST0000: new disk, initializing [ 6974.342321] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6974.366866] Lustre: 436422:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.103@tcp: can't add NID 192.168.201.103@tcp130: rc = -75 [ 6974.370374] Lustre: 436422:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [ 6974.439060] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6976.881811] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6979.567633] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6979.575514] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6979.592217] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6980.390054] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6984.674346] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6984.675504] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6984.677631] LustreError: Skipped 1 previous similar message [ 6984.684111] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6984.688122] Lustre: Skipped 1 previous similar message [ 6989.201722] Lustre: server umount lustre-OST0000 complete [ 6991.008889] LNetError: 434677:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.103 [ 6991.011719] LNetError: 434677:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Skipped 3 previous similar messages [ 6991.013543] LNetError: Protocol error connecting to 192.168.201.103@tcp120 at host 192.168.201.103:988: is it running a compatible version of Lustre? [ 6991.840927] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6991.843478] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6991.846823] Lustre: Skipped 1 previous similar message [ 6991.848396] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6994.913115] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6994.913479] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6995.104989] LNetError: 434685:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.103 [ 6995.107869] LNetError: Protocol error connecting to 192.168.201.103@tcp121 at host 192.168.201.103:988: is it running a compatible version of Lustre? [ 6996.571720] Lustre: server umount lustre-MDT0000 complete [ 6997.785288] LustreError: 434664:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772685235 with bad export cookie 1228131508869124675 [ 6997.788142] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6997.788623] LustreError: 434664:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6997.901899] Lustre: server umount lustre-MDT0001 complete [ 7000.225193] LNetError: 434675:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.103 [ 7000.228338] LNetError: 434675:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Skipped 1 previous similar message [ 7000.230718] LNetError: Protocol error connecting to 192.168.201.103@tcp122 at host 192.168.201.103:988: is it running a compatible version of Lustre? [ 7000.234136] LNetError: Skipped 1 previous similar message [ 7001.440753] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 7002.427327] Key type lgssc unregistered [ 7002.589723] LNet: 438205:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7002.593532] LNetError: 438205:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7002.596873] LNet: Removed LNI 192.168.201.103@tcp [ 7003.616170] LNet: Removed LNI 192.168.201.103@tcp100 [ 7003.618326] LNetError: 438205:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7003.622386] LNetError: 438205:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 7003.625622] LNet: Removed LNI 192.168.201.103@tcp101 [ 7004.640141] LNet: Removed LNI 192.168.201.103@tcp102 [ 7004.641433] LNetError: 438205:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7004.643860] LNetError: 438205:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 7004.647439] LNet: Removed LNI 192.168.201.103@tcp103 [ 7005.664114] LNet: Removed LNI 192.168.201.103@tcp104 [ 7006.688104] LNet: Removed LNI 192.168.201.103@tcp105 [ 7006.689405] LNetError: 438205:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7006.691354] LNetError: 438205:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 7006.694855] LNet: Removed LNI 192.168.201.103@tcp106 [ 7007.712111] LNet: Removed LNI 192.168.201.103@tcp107 [ 7008.736169] LNet: Removed LNI 192.168.201.103@tcp108 [ 7009.760195] LNet: Removed LNI 192.168.201.103@tcp109 [ 7010.784162] LNet: Removed LNI 192.168.201.103@tcp110 [ 7010.786221] LNetError: 438205:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7010.788039] LNetError: 438205:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 7010.789987] LNet: Removed LNI 192.168.201.103@tcp111 [ 7011.808126] LNet: Removed LNI 192.168.201.103@tcp112 [ 7012.832142] LNet: Removed LNI 192.168.201.103@tcp113 [ 7013.856105] LNet: Removed LNI 192.168.201.103@tcp114 [ 7014.880189] LNet: Removed LNI 192.168.201.103@tcp115 [ 7015.904091] LNet: Removed LNI 192.168.201.103@tcp116 [ 7016.928129] LNet: Removed LNI 192.168.201.103@tcp117 [ 7017.952122] LNet: Removed LNI 192.168.201.103@tcp118 [ 7018.976108] LNet: Removed LNI 192.168.201.103@tcp119 [ 7018.977310] LNetError: 438205:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7018.979288] LNetError: 438205:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [ 7018.981382] LNet: Removed LNI 192.168.201.103@tcp120 [ 7020.000147] LNet: Removed LNI 192.168.201.103@tcp121 [ 7021.024148] LNet: Removed LNI 192.168.201.103@tcp122 [ 7021.026079] LNet: Removed LNI 192.168.201.103@tcp123 [ 7021.027853] LNet: Removed LNI 192.168.201.103@tcp124 [ 7021.029256] LNet: Removed LNI 192.168.201.103@tcp125 [ 7021.030240] LNet: Removed LNI 192.168.201.103@tcp126 [ 7021.031260] LNet: Removed LNI 192.168.201.103@tcp127 [ 7021.032242] LNet: Removed LNI 192.168.201.103@tcp128 [ 7021.033204] LNet: Removed LNI 192.168.201.103@tcp129 [ 7021.034196] LNet: Removed LNI 192.168.201.103@tcp130 [ 7021.035187] LNet: Removed LNI 192.168.201.103@tcp131 [ 7021.036112] LNet: Removed LNI 192.168.201.103@tcp132 [ 7021.037086] LNet: Removed LNI 192.168.201.103@tcp133 [ 7021.038080] LNet: Removed LNI 192.168.201.103@tcp134 [ 7021.045413] LNet: Removed LNI 192.168.201.103@tcp135 [ 7021.310643] Key type .llcrypt unregistered [ 7021.311536] Key type ._llcrypt unregistered [ 7028.256784] Key type ._llcrypt registered [ 7028.258195] Key type .llcrypt registered [ 7028.298282] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 7034.218820] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 7034.594195] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7034.630277] alg: No test for adler32 (adler32-zlib) [ 7035.487179] Lustre: Lustre: Build Version: 2.17.50_193_g407ab0b [ 7035.573282] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 7037.152112] Key type lgssc registered [ 7037.463807] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7047.883235] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 23:34:46 (1772685286) [ 7053.488541] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 7054.487216] Key type lgssc unregistered [ 7054.612369] LNet: 442212:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7054.614412] LNetError: 442212:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7054.622344] LNet: Removed LNI 192.168.201.103@tcp [ 7054.906392] Key type .llcrypt unregistered [ 7054.907339] Key type ._llcrypt unregistered [ 7061.368333] Key type ._llcrypt registered [ 7061.369523] Key type .llcrypt registered [ 7061.409502] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 7061.853439] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7061.867541] alg: No test for adler32 (adler32-zlib) [ 7062.732740] Lustre: Lustre: Build Version: 2.17.50_193_g407ab0b [ 7062.817678] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 7064.392174] Key type lgssc registered [ 7064.741917] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7067.767198] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7067.772186] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7068.860886] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7068.871307] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7068.910789] Lustre: lustre-MDT0000: new disk, initializing [ 7068.936264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7068.944593] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7070.291857] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7074.325193] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7074.371978] Lustre: 443383:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7074.399656] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7074.401470] Lustre: Skipped 1 previous similar message [ 7074.438078] Lustre: lustre-MDT0001: new disk, initializing [ 7074.472731] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7074.485094] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7074.488602] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7075.743834] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7078.735301] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7079.909732] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7082.090341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7082.187315] Lustre: lustre-OST0000: new disk, initializing [ 7082.189780] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7082.218885] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7083.956823] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7086.957405] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7088.119456] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing lnet_if_list [ 7089.142164] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7089.146191] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7089.177447] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7089.702785] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [ 7089.713524] LNet: Added LNI 192.168.201.103@tcp10 [8/256/0/180] [ 7091.580676] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [ 7091.590714] LNet: Added LNI 192.168.201.103@tcp11 [8/256/0/180] [ 7092.984160] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [ 7092.994819] LNet: Added LNI 192.168.201.103@tcp12 [8/256/0/180] [ 7094.331071] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [ 7094.340870] LNet: Added LNI 192.168.201.103@tcp13 [8/256/0/180] [ 7095.722669] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [ 7095.734086] LNet: Added LNI 192.168.201.103@tcp14 [8/256/0/180] [ 7097.066313] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [ 7097.076484] LNet: Added LNI 192.168.201.103@tcp15 [8/256/0/180] [ 7098.421124] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [ 7098.430638] LNet: Added LNI 192.168.201.103@tcp16 [8/256/0/180] [ 7099.855684] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [ 7099.866476] LNet: Added LNI 192.168.201.103@tcp17 [8/256/0/180] [ 7101.364383] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [ 7101.376912] LNet: Added LNI 192.168.201.103@tcp18 [8/256/0/180] [ 7102.853194] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [ 7102.863609] LNet: Added LNI 192.168.201.103@tcp19 [8/256/0/180] [ 7104.426468] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [ 7104.437725] LNet: Added LNI 192.168.201.103@tcp20 [8/256/0/180] [ 7106.007763] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [ 7106.017698] LNet: Added LNI 192.168.201.103@tcp21 [8/256/0/180] [ 7107.506942] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [ 7107.516182] LNet: Added LNI 192.168.201.103@tcp22 [8/256/0/180] [ 7109.101303] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [ 7109.114034] LNet: Added LNI 192.168.201.103@tcp23 [8/256/0/180] [ 7110.694269] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [ 7110.704410] LNet: Added LNI 192.168.201.103@tcp24 [8/256/0/180] [ 7112.329572] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [ 7112.340304] LNet: Added LNI 192.168.201.103@tcp25 [8/256/0/180] [ 7114.009545] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [ 7114.019645] LNet: Added LNI 192.168.201.103@tcp26 [8/256/0/180] [ 7115.951375] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [ 7115.961918] LNet: Added LNI 192.168.201.103@tcp27 [8/256/0/180] [ 7117.962291] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [ 7117.973905] LNet: Added LNI 192.168.201.103@tcp28 [8/256/0/180] [ 7119.985385] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [ 7119.998474] LNet: Added LNI 192.168.201.103@tcp29 [8/256/0/180] [ 7121.971303] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [ 7121.981099] LNet: Added LNI 192.168.201.103@tcp30 [8/256/0/180] [ 7123.843100] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [ 7123.852306] LNet: Added LNI 192.168.201.103@tcp31 [8/256/0/180] [ 7125.748203] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [ 7125.758502] LNet: Added LNI 192.168.201.103@tcp32 [8/256/0/180] [ 7127.661427] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [ 7127.674284] LNet: Added LNI 192.168.201.103@tcp33 [8/256/0/180] [ 7129.615522] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [ 7129.625818] LNet: Added LNI 192.168.201.103@tcp34 [8/256/0/180] [ 7131.632737] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [ 7131.643336] LNet: Added LNI 192.168.201.103@tcp35 [8/256/0/180] [ 7133.631816] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [ 7133.641881] LNet: Added LNI 192.168.201.103@tcp36 [8/256/0/180] [ 7135.630642] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [ 7135.641446] LNet: Added LNI 192.168.201.103@tcp37 [8/256/0/180] [ 7137.592222] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [ 7137.604095] LNet: Added LNI 192.168.201.103@tcp38 [8/256/0/180] [ 7139.621995] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [ 7139.637383] LNet: Added LNI 192.168.201.103@tcp39 [8/256/0/180] [ 7141.736952] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [ 7141.748281] LNet: Added LNI 192.168.201.103@tcp40 [8/256/0/180] [ 7143.801310] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [ 7143.812516] LNet: Added LNI 192.168.201.103@tcp41 [8/256/0/180] [ 7146.052650] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 7146.063186] LNet: Added LNI 192.168.201.103@tcp42 [8/256/0/180] [ 7148.142442] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [ 7148.153460] LNet: Added LNI 192.168.201.103@tcp43 [8/256/0/180] [ 7150.300823] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [ 7150.310382] LNet: Added LNI 192.168.201.103@tcp44 [8/256/0/180] [ 7152.498482] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [ 7152.507617] LNet: Added LNI 192.168.201.103@tcp45 [8/256/0/180] [ 7154.894333] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [ 7154.903027] LNet: Added LNI 192.168.201.103@tcp46 [8/256/0/180] [ 7157.198492] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [ 7157.207850] LNet: Added LNI 192.168.201.103@tcp47 [8/256/0/180] [ 7159.551369] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [ 7159.560876] LNet: Added LNI 192.168.201.103@tcp48 [8/256/0/180] [ 7162.048835] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [ 7162.057931] LNet: Added LNI 192.168.201.103@tcp49 [8/256/0/180] [ 7164.417957] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [ 7164.428228] LNet: Added LNI 192.168.201.103@tcp50 [8/256/0/180] [ 7172.065106] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7172.067030] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7172.071407] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7176.162366] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7176.162963] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7176.165323] Lustre: Skipped 1 previous similar message [ 7176.166873] Lustre: Skipped 1 previous similar message [ 7177.563320] Lustre: server umount lustre-MDT0000 complete [ 7178.678063] LustreError: 443377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772685416 with bad export cookie 2105644989830782413 [ 7178.680971] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7178.684374] LustreError: 443377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7178.790842] Lustre: server umount lustre-MDT0001 complete [ 7185.536274] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 7188.527738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7197.664143] Lustre: 442660:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772685419/real 1772685419] req@0000000027d1fb15 x1858795261211520/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1772685435 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7197.669764] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7199.712122] Lustre: 442658:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772685421/real 1772685421] req@0000000099c3c94f x1858795261211648/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1772685437 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7202.784112] Lustre: 442657:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772685424/real 1772685424] req@00000000da668206 x1858795261211776/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1772685440 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7203.808876] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x1d38c0f649f1e5ad [ 7203.811060] Lustre: MGC192.168.201.103@tcp: Connection restored to 0@lo (at 0@lo) [ 7203.921861] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7204.960090] Lustre: 442659:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772685426/real 1772685426] req@00000000af5b0d30 x1858795261212032/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1772685442 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7205.186529] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7207.813025] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7207.941173] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7209.000754] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7209.235293] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7210.016131] Lustre: 442660:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772685431/real 1772685431] req@000000008ab25b35 x1858795261212160/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1772685447 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7210.924033] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7212.065297] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7217.193631] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7219.169588] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7219.171671] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7219.175223] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7219.176679] Lustre: Skipped 1 previous similar message [ 7222.244426] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7224.977777] Lustre: server umount lustre-OST0000 complete [ 7227.362753] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7227.363555] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7227.365888] Lustre: Skipped 2 previous similar messages [ 7227.370833] Lustre: Skipped 2 previous similar messages [ 7232.279396] Lustre: server umount lustre-MDT0000 complete [ 7232.480905] LustreError: 468511:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7232.485765] LustreError: 468511:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7233.514548] LustreError: 461871:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772685471 with bad export cookie 2105644989830784429 [ 7233.517965] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7233.518111] LustreError: 461871:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7233.650309] Lustre: server umount lustre-MDT0001 complete [ 7236.934542] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 7237.943675] Key type lgssc unregistered [ 7238.109740] LNet: 471041:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7238.112907] LNetError: 471041:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7238.114828] LNet: Removed LNI 192.168.201.103@tcp [ 7238.116206] LNet: Removed LNI 192.168.201.103@tcp10 [ 7238.117424] LNet: Removed LNI 192.168.201.103@tcp11 [ 7238.118578] LNet: Removed LNI 192.168.201.103@tcp12 [ 7238.119750] LNet: Removed LNI 192.168.201.103@tcp13 [ 7238.120806] LNet: Removed LNI 192.168.201.103@tcp14 [ 7238.121805] LNet: Removed LNI 192.168.201.103@tcp15 [ 7238.122861] LNet: Removed LNI 192.168.201.103@tcp16 [ 7238.123946] LNet: Removed LNI 192.168.201.103@tcp17 [ 7238.124878] LNet: Removed LNI 192.168.201.103@tcp18 [ 7238.125883] LNet: Removed LNI 192.168.201.103@tcp19 [ 7238.126878] LNet: Removed LNI 192.168.201.103@tcp20 [ 7238.127898] LNet: Removed LNI 192.168.201.103@tcp21 [ 7238.128890] LNet: Removed LNI 192.168.201.103@tcp22 [ 7238.129863] LNet: Removed LNI 192.168.201.103@tcp23 [ 7238.130902] LNet: Removed LNI 192.168.201.103@tcp24 [ 7238.131880] LNet: Removed LNI 192.168.201.103@tcp25 [ 7238.132878] LNet: Removed LNI 192.168.201.103@tcp26 [ 7238.134025] LNet: Removed LNI 192.168.201.103@tcp27 [ 7238.135091] LNet: Removed LNI 192.168.201.103@tcp28 [ 7238.136412] LNet: Removed LNI 192.168.201.103@tcp29 [ 7238.137417] LNet: Removed LNI 192.168.201.103@tcp30 [ 7238.138580] LNet: Removed LNI 192.168.201.103@tcp31 [ 7238.139668] LNet: Removed LNI 192.168.201.103@tcp32 [ 7238.140618] LNet: Removed LNI 192.168.201.103@tcp33 [ 7238.141683] LNet: Removed LNI 192.168.201.103@tcp34 [ 7238.142740] LNet: Removed LNI 192.168.201.103@tcp35 [ 7238.143736] LNet: Removed LNI 192.168.201.103@tcp36 [ 7238.144710] LNet: Removed LNI 192.168.201.103@tcp37 [ 7238.145853] LNet: Removed LNI 192.168.201.103@tcp38 [ 7238.146898] LNet: Removed LNI 192.168.201.103@tcp39 [ 7238.147831] LNet: Removed LNI 192.168.201.103@tcp40 [ 7238.148885] LNet: Removed LNI 192.168.201.103@tcp41 [ 7238.149824] LNet: Removed LNI 192.168.201.103@tcp42 [ 7238.150793] LNet: Removed LNI 192.168.201.103@tcp43 [ 7238.151837] LNet: Removed LNI 192.168.201.103@tcp44 [ 7238.152762] LNet: Removed LNI 192.168.201.103@tcp45 [ 7238.153686] LNet: Removed LNI 192.168.201.103@tcp46 [ 7238.154653] LNet: Removed LNI 192.168.201.103@tcp47 [ 7238.155625] LNet: Removed LNI 192.168.201.103@tcp48 [ 7238.156661] LNet: Removed LNI 192.168.201.103@tcp49 [ 7238.163335] LNet: Removed LNI 192.168.201.103@tcp50 [ 7238.432325] Key type .llcrypt unregistered [ 7238.433398] Key type ._llcrypt unregistered [ 7245.175941] Key type ._llcrypt registered [ 7245.177015] Key type .llcrypt registered [ 7245.216644] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 7245.548568] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7245.556109] alg: No test for adler32 (adler32-zlib) [ 7246.425579] Lustre: Lustre: Build Version: 2.17.50_193_g407ab0b [ 7246.507605] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 7248.088171] Key type lgssc registered [ 7248.422316] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7250.774711] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7250.779623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7251.932807] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7253.114358] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7255.781299] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7255.899383] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7257.073156] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7258.725293] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7259.917166] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7261.971582] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7262.081515] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7263.144890] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 7263.855787] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7265.486878] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7269.875986] loop0: detected capacity change from 0 to 204800000 [ 7269.945304] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7269.948301] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7269.951289] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7271.808084] loop0: detected capacity change from 0 to 204800000 [ 7271.870948] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7271.876933] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7271.882540] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7273.539620] loop0: detected capacity change from 0 to 204800000 [ 7275.350598] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7275.403315] Lustre: fs15246-OST0000: new disk, initializing [ 7275.405355] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 7275.423906] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7277.265200] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7281.046210] loop1: detected capacity change from 0 to 204800000 [ 7282.902249] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7282.941112] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 7282.966391] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 7283.002546] Lustre: fs15246-MDT0000: new disk, initializing [ 7283.048763] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7283.056583] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7283.096326] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 7284.128724] Lustre: 476620:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@000000006df6a175 x1858795454100096/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1772685537 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 7284.235467] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7293.216316] LustreError: 476620:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 7293.223057] LustreError: 476620:0:(fid_request.c:255:seq_client_get_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Can't allocate new sequence: rc = -5 [ 7293.228577] LustreError: 476620:0:(osp_precreate.c:459:osp_precreate_rollover_new_seq()) fs15246-OST0000-osc-MDT0000: alloc fid error: rc = -5 [ 7294.436264] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7294.440739] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 7297.248541] Lustre: server umount fs15246-MDT0000 complete [ 7299.556066] LustreError: 472258:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7299.560920] LustreError: 472258:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7300.643900] Lustre: server umount fs15246-OST0000 complete [ 7304.161122] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7304.165616] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7304.173805] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7309.794174] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7309.795706] Lustre: Skipped 2 previous similar messages [ 7310.224053] Lustre: server umount lustre-OST0000 complete [ 7312.352832] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7312.355272] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7312.358846] Lustre: Skipped 1 previous similar message [ 7312.360685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7314.914130] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7317.544817] Lustre: server umount lustre-MDT0000 complete [ 7319.071361] LustreError: 472243:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772685556 with bad export cookie 15659368390359628074 [ 7319.075069] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7319.075267] LustreError: 472243:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7319.190611] Lustre: server umount lustre-MDT0001 complete [ 7322.638528] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 7323.863707] Key type lgssc unregistered [ 7323.980341] LNet: 479682:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7323.983244] LNetError: 479682:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7323.992382] LNet: Removed LNI 192.168.201.103@tcp [ 7324.410118] Key type .llcrypt unregistered [ 7324.412263] Key type ._llcrypt unregistered [ 7332.983508] Key type ._llcrypt registered [ 7332.985698] Key type .llcrypt registered [ 7333.059392] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 7333.491471] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7333.535104] alg: No test for adler32 (adler32-zlib) [ 7334.459426] Lustre: Lustre: Build Version: 2.17.50_193_g407ab0b [ 7334.560155] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 7336.160327] Key type lgssc registered [ 7336.600650] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7348.761859] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 7351.822804] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 7364.802361] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7364.808893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7365.909753] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7365.920431] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7365.957670] Lustre: lustre-MDT0000: new disk, initializing [ 7365.983272] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7365.990838] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7367.225887] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7371.838227] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7371.881895] Lustre: 484065:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7371.915497] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7371.917407] Lustre: Skipped 1 previous similar message [ 7371.961671] Lustre: lustre-MDT0001: new disk, initializing [ 7372.016433] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7372.035119] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7372.040829] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7373.513208] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7376.536494] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7377.773569] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7380.550286] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7380.674422] Lustre: lustre-OST0000: new disk, initializing [ 7380.676596] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7380.709523] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7382.890823] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7383.173978] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7383.177802] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7383.224841] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7385.948468] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7387.297934] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7387.379806] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7388.677061] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7388.753850] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7391.585473] Lustre: server umount lustre-OST0000 complete [ 7393.763064] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7393.763750] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7393.766693] Lustre: Skipped 1 previous similar message [ 7393.769079] Lustre: Skipped 1 previous similar message [ 7398.881524] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7398.883257] Lustre: Skipped 1 previous similar message [ 7403.005308] Lustre: server umount lustre-MDT0000 complete [ 7404.186476] LustreError: 484059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772685642 with bad export cookie 2390761456483862291 [ 7404.190274] LustreError: 484059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7404.191070] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7414.807653] Lustre: server umount lustre-MDT0001 complete [ 7422.376912] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 7425.619491] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 7435.873857] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7269 sec ========== 23:41:14 (1772685674) [ 7436.414453] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 23:41:14 (1772685674) === [ 7437.626937] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 23:41:15 (1772685675) === [ 7451.186755] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 7452.262782] Key type lgssc unregistered [ 7452.407481] LNet: 492245:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7452.410395] LNetError: 492245:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7452.418620] LNet: Removed LNI 192.168.201.103@tcp [ 7452.770687] Key type .llcrypt unregistered [ 7452.771641] Key type ._llcrypt unregistered