[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 403500734 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002156] x2apic enabled [ 0.003004] Switched APIC routing to physical x2apic. [ 0.003910] kvm-guest: setup PV IPIs [ 0.005839] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.006000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.006009] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.007005] pid_max: default: 32768 minimum: 301 [ 0.008068] LSM: Security Framework initializing [ 0.009024] Yama: becoming mindful. [ 0.009685] SELinux: Initializing. [ 0.010037] *** VALIDATE selinux *** [ 0.017177] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.021581] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.022078] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.023059] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.024068] *** VALIDATE tmpfs *** [ 0.025241] *** VALIDATE proc *** [ 0.026018] *** VALIDATE cgroup *** [ 0.026679] *** VALIDATE cgroup2 *** [ 0.027172] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.028095] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.029003] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.030016] Spectre V2 : User space: Vulnerable [ 0.031003] Speculative Store Bypass: Vulnerable [ 0.034098] debug: unmapping init [mem 0xffffffff9b059000-0xffffffff9b060fff] [ 0.036155] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.037395] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.038010] ... version: 2 [ 0.038742] ... bit width: 48 [ 0.039006] ... generic registers: 4 [ 0.039867] ... value mask: 0000ffffffffffff [ 0.040005] ... max period: 00007fffffffffff [ 0.041006] ... fixed-purpose events: 3 [ 0.041757] ... event mask: 000000070000000f [ 0.042212] rcu: Hierarchical SRCU implementation. [ 0.044511] smp: Bringing up secondary CPUs ... [ 0.045347] x86: Booting SMP configuration: [ 0.046015] .... node #0, CPUs: #1 #2 #3 [ 0.049390] smp: Brought up 1 node, 4 CPUs [ 0.051010] smpboot: Max logical packages: 1 [ 0.052010] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.234023] node 0 deferred pages initialised in 181ms [ 0.236107] devtmpfs: initialized [ 0.237131] x86/mm: Memory block size: 128MB [ 0.239091] gcov: version magic: 0x41383552 [ 0.240440] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.241050] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.242150] pinctrl core: initialized pinctrl subsystem [ 0.243090] [ 0.243437] ************************************************************* [ 0.244010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.245005] ** ** [ 0.246006] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.247006] ** ** [ 0.248006] ** This means that this kernel is built to expose internal ** [ 0.249006] ** IOMMU data structures, which may compromise security on ** [ 0.250006] ** your system. ** [ 0.251005] ** ** [ 0.252005] ** If you see this message and you are not debugging the ** [ 0.253006] ** kernel, report this immediately to your vendor! ** [ 0.254006] ** ** [ 0.255006] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.256005] ************************************************************* [ 0.257399] NET: Registered protocol family 16 [ 0.258286] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.259025] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.260026] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.261248] cpuidle: using governor menu [ 0.262469] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.265432] PCI: Using configuration type 1 for base access [ 0.267101] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.274072] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.275011] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.276083] cryptd: max_cpu_qlen set to 1000 [ 0.277162] ACPI: Added _OSI(Module Device) [ 0.279009] ACPI: Added _OSI(Processor Device) [ 0.280009] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.282007] ACPI: Added _OSI(Processor Aggregator Device) [ 0.285539] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.292520] ACPI: Interpreter enabled [ 0.294063] ACPI: PM: (supports S0 S3 S4 S5) [ 0.296012] ACPI: Using IOAPIC for interrupt routing [ 0.297163] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.301439] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.313098] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.315021] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.317010] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.318035] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.320895] acpiphp: Slot [2] registered [ 0.322076] acpiphp: Slot [5] registered [ 0.322876] acpiphp: Slot [6] registered [ 0.324061] acpiphp: Slot [7] registered [ 0.324844] acpiphp: Slot [8] registered [ 0.325093] acpiphp: Slot [9] registered [ 0.327094] acpiphp: Slot [10] registered [ 0.328063] acpiphp: Slot [3] registered [ 0.329030] acpiphp: Slot [4] registered [ 0.329876] acpiphp: Slot [11] registered [ 0.331045] acpiphp: Slot [12] registered [ 0.331911] acpiphp: Slot [13] registered [ 0.332056] acpiphp: Slot [14] registered [ 0.332869] acpiphp: Slot [15] registered [ 0.334046] acpiphp: Slot [16] registered [ 0.334925] acpiphp: Slot [17] registered [ 0.336044] acpiphp: Slot [18] registered [ 0.336910] acpiphp: Slot [19] registered [ 0.337055] acpiphp: Slot [20] registered [ 0.337882] acpiphp: Slot [21] registered [ 0.339045] acpiphp: Slot [22] registered [ 0.339809] acpiphp: Slot [23] registered [ 0.341092] acpiphp: Slot [24] registered [ 0.342069] acpiphp: Slot [25] registered [ 0.344053] acpiphp: Slot [26] registered [ 0.345073] acpiphp: Slot [27] registered [ 0.346055] acpiphp: Slot [28] registered [ 0.347056] acpiphp: Slot [29] registered [ 0.348059] acpiphp: Slot [30] registered [ 0.350107] acpiphp: Slot [31] registered [ 0.351051] PCI host bridge to bus 0000:00 [ 0.352012] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.354009] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.356011] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.358010] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.361013] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.363013] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.365118] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.366763] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.369872] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.377008] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.380316] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.383010] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.385008] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.387011] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.389604] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.391790] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.395027] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.397679] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.403011] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.415011] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.421008] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.426892] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.431009] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.435007] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.446008] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.451916] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.456007] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.460007] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.470009] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.477848] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.481008] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.485008] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.494010] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.501830] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.505008] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.510008] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.518008] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.525571] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.529008] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.533007] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.543009] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.550064] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.553008] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.557008] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.564008] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.571992] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.573244] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.575189] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.576216] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.578107] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.581132] iommu: Default domain type: Passthrough [ 0.582296] SCSI subsystem initialized [ 0.583073] ACPI: bus type USB registered [ 0.583973] usbcore: registered new interface driver usbfs [ 0.585030] usbcore: registered new interface driver hub [ 0.586035] usbcore: registered new device driver usb [ 0.587072] pps_core: LinuxPPS API ver. 1 registered [ 0.588006] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.590030] PTP clock support registered [ 0.591076] EDAC MC: Ver: 3.0.0 [ 0.592076] PCI: Using ACPI for IRQ routing [ 0.593579] NetLabel: Initializing [ 0.594006] NetLabel: domain hash size = 128 [ 0.594838] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.596042] NetLabel: unlabeled traffic allowed by default [ 0.598025] vgaarb: loaded [ 0.598594] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.599005] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.604272] clocksource: Switched to clocksource kvm-clock [ 0.686488] VFS: Disk quotas dquot_6.6.0 [ 0.687382] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.688623] *** VALIDATE ramfs *** [ 0.689278] *** VALIDATE hugetlbfs *** [ 0.690096] pnp: PnP ACPI init [ 0.691498] pnp: PnP ACPI: found 6 devices [ 0.704298] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.705912] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.707005] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.708114] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.709295] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.710491] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.711974] NET: Registered protocol family 2 [ 0.713325] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.716162] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.717991] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.721403] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.723281] TCP: Hash tables configured (established 65536 bind 65536) [ 0.724732] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.726315] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.727797] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.729270] NET: Registered protocol family 1 [ 0.730617] RPC: Registered named UNIX socket transport module. [ 0.731696] RPC: Registered udp transport module. [ 0.732546] RPC: Registered tcp transport module. [ 0.733411] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.734609] NET: Registered protocol family 44 [ 0.735516] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.736662] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.737803] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.739088] PCI: CLS 0 bytes, default 64 [ 0.740088] Unpacking initramfs... [ 1.854612] debug: unmapping init [mem 0xffff9af8fcc54000-0xffff9af8fffbffff] [ 1.857026] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 1.858304] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 1.859795] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 2.277776] Initialise system trusted keyrings [ 2.278684] Key type blacklist registered [ 2.279743] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.285678] zbud: loaded [ 2.287519] *** VALIDATE nfs *** [ 2.288215] *** VALIDATE nfs4 *** [ 2.289083] pstore: using deflate compression [ 2.290656] Platform Keyring initialized [ 2.352854] NET: Registered protocol family 38 [ 2.353913] Key type asymmetric registered [ 2.354788] Asymmetric key parser 'x509' registered [ 2.355931] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.357688] io scheduler mq-deadline registered [ 2.358628] io scheduler kyber registered [ 2.359776] io scheduler bfq registered [ 2.360827] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.362505] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.364233] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.366197] ACPI: Power Button [PWRF] [ 2.369496] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.373677] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.380311] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.384257] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.392306] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.417023] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.442244] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.445269] Non-volatile memory driver v1.3 [ 2.446197] Linux agpgart interface v0.103 [ 2.472905] virtio_blk virtio1: [vda] 134736 512-byte logical blocks (69.0 MB/65.8 MiB) [ 2.476378] vda: detected capacity change from 0 to 68984832 [ 2.490362] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.492264] vdb: detected capacity change from 0 to 1073741824 [ 2.501340] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.502870] vdc: detected capacity change from 0 to 2621440000 [ 2.512334] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.513940] vdd: detected capacity change from 0 to 2621440000 [ 2.523827] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.525446] vde: detected capacity change from 0 to 4294967296 [ 2.535174] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.536819] vdf: detected capacity change from 0 to 4294967296 [ 2.540500] libphy: Fixed MDIO Bus: probed [ 2.543457] usbcore: registered new interface driver usbserial_generic [ 2.544808] usbserial: USB Serial support registered for generic [ 2.546105] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.548351] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.549281] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.550523] mousedev: PS/2 mouse device common for all mice [ 2.552087] rtc_cmos 00:05: RTC can wake from S4 [ 2.553498] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.555539] rtc_cmos 00:05: registered as rtc0 [ 2.556599] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.557079] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.559237] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.560213] intel_pstate: CPU model not supported [ 2.563589] hid: raw HID events driver (C) Jiri Kosina [ 2.564727] usbcore: registered new interface driver usbhid [ 2.565931] usbhid: USB HID core driver [ 2.566723] drop_monitor: Initializing network drop monitor service [ 2.568074] Initializing XFRM netlink socket [ 2.569272] NET: Registered protocol family 10 [ 2.571178] Segment Routing with IPv6 [ 2.571927] NET: Registered protocol family 17 [ 2.573026] mpls_gso: MPLS GSO support [ 2.577195] RAS: Correctable Errors collector initialized. [ 2.578542] AVX version of gcm_enc/dec engaged. [ 2.579484] AES CTR mode by8 optimization enabled [ 2.627052] sched_clock: Marking stable (2627027685, 0)->(3284159686, -657132001) [ 2.629021] registered taskstats version 1 [ 2.630139] Loading compiled-in X.509 certificates [ 2.631385] zswap: loaded using pool lzo/zbud [ 2.651582] Key type big_key registered [ 2.662941] Key type encrypted registered [ 2.664482] ima: No TPM chip found, activating TPM-bypass! [ 2.666625] ima: Allocated hash algorithm: sha1 [ 2.668558] ima: No architecture policies found [ 2.669843] evm: Initialising EVM extended attributes: [ 2.671922] evm: security.selinux [ 2.673273] evm: security.ima [ 2.674620] evm: security.capability [ 2.676174] evm: HMAC attrs: 0x1 [ 2.678973] rtc_cmos 00:05: setting system clock to 2026-02-27 04:41:16 UTC (1772167276) [ 2.686206] debug: unmapping init [mem 0xffffffff9c003000-0xffffffff9c1fffff] [ 2.688987] debug: unmapping init [mem 0xffffffff9ad82000-0xffffffff9b058fff] [ 2.696110] Write protecting the kernel read-only data: 28672k [ 2.698768] debug: unmapping init [mem 0xffffffff99403000-0xffffffff995fffff] [ 2.700398] debug: unmapping init [mem 0xffffffff99d14000-0xffffffff99dfffff] [ 2.728229] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 2.733411] systemd[1]: Detected virtualization kvm. [ 2.734702] systemd[1]: Detected architecture x86-64. [ 2.735902] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 2.759770] systemd[1]: No hostname configured. [ 2.760971] systemd[1]: Set hostname to . [ 2.762253] random: systemd: uninitialized urandom read (16 bytes read) [ 2.763739] systemd[1]: Initializing machine ID from random generator. [ 2.794902] random: ln: uninitialized urandom read (6 bytes read) [ 2.889830] random: systemd: uninitialized urandom read (16 bytes read) [ 2.892578] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 2.896721] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 2.899763] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Slices. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Initrd Root Device. Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Sockets. Starting Apply Kernel Variables... Starting Setup Virtual Console... Starting Journal Service... [ OK ] Reached target Swap. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.413564] device-mapper: uevent: version 1.0.3 [ 3.414850] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.154397] random: fast init done [ 4.174732] virtio_net virtio0 ens2: renamed from eth0 [ 4.186417] scsi host0: ata_piix [ 4.207793] scsi host1: ata_piix [ 4.209577] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.212604] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 7.820737] dracut-initqueue[578]: RTNETLINK answers: File exists [ 9.181335] random: crng init done [ 9.182827] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 9.431468] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 10.325639] printk: systemd: 26 output lines suppressed due to ratelimiting [ 10.470250] SELinux: Disabled at runtime. [ 10.518484] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 10.524341] systemd[1]: Detected virtualization kvm. [ 10.525416] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 10.872870] systemd[1]: initrd-switch-root.service: Succeeded. [ 10.876220] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 10.880421] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 10.884071] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 10.887298] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 10.894524] systemd[1]: Starting Journal Service... Starting Journal Service... [ 10.900983] systemd[1]: Starting Remount Root and Kernel File Systems... Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-getty.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting POSIX Message Queue File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Huge Pages File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target rpc_pipefs.target. Starting Apply Kernel Variables... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ 11.002414] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Reached target Local Encrypted Volumes. Mounting Kernel Debug File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 11.346530] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 11.613880] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 11.635993] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 11.860321] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 11.869461] EDAC sbridge: Ver: 1.1.2 [ 13.183598] Key type dns_resolver registered [ 13.455177] NFS: Registering the id_resolver key type [ 13.456678] Key type id_resolver registered [ 13.457797] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Crash recovery kernel arming. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg454-server login: [ 38.116845] libcfs: loading out-of-tree module taints kernel. [ 38.133488] Key type ._llcrypt registered [ 38.134363] Key type .llcrypt registered [ 38.166276] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 45.154631] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 45.669595] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 45.674307] alg: No test for adler32 (adler32-zlib) [ 46.620395] Lustre: Lustre: Build Version: 2.17.50_152_g82f9dcf [ 46.882297] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 48.479160] Key type lgssc registered [ 49.026463] Lustre: Echo OBD driver; http://www.lustre.org/ [ 55.382568] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 70.690421] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 75.597612] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 75.604647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 76.704088] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 76.716418] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 76.753148] Lustre: lustre-MDT0000: new disk, initializing [ 76.778270] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 76.785655] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 78.190792] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 84.422369] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 84.469778] Lustre: 6469:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 84.485105] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 84.487297] Lustre: Skipped 1 previous similar message [ 84.528203] Lustre: lustre-MDT0001: new disk, initializing [ 84.549657] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 84.556734] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 84.559866] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 85.905326] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 88.416895] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 92.014720] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 92.151431] Lustre: lustre-OST0000: new disk, initializing [ 92.156332] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 92.183342] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 92.521587] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 92.526160] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 92.555440] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 94.374460] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 101.482324] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 101.546355] Lustre: lustre-OST0001: new disk, initializing [ 101.549498] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 101.574438] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 103.767511] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 107.505896] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 107.511439] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 107.526933] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 110.008805] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 117.291415] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 124.816573] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing check_logdir /tmp/testlogs/ [ 126.292837] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing yml_node [ 128.203193] Lustre: DEBUG MARKER: Client: 2.17.50.152 [ 129.386610] Lustre: DEBUG MARKER: MDS: 2.17.50.152 [ 130.535984] Lustre: DEBUG MARKER: OSS: 2.17.50.152 [ 131.294637] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Feb 26 23:43:25 EST 2026 [ 139.621451] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 140.087350] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 153.568151] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 153.569484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 153.575664] Lustre: Skipped 3 previous similar messages [ 153.579318] Lustre: Skipped 2 previous similar messages [ 158.687783] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 158.689978] Lustre: Skipped 4 previous similar messages [ 163.808264] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 163.812332] Lustre: Skipped 3 previous similar messages [ 166.367233] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 166.452314] Lustre: server umount lustre-MDT0000 complete [ 168.033575] LustreError: 6460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772167441 with bad export cookie 8254837337354197294 [ 168.034951] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 168.038911] LustreError: 6460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 168.928176] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 168.928626] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 168.935401] Lustre: Skipped 1 previous similar message [ 168.941351] Lustre: Skipped 1 previous similar message [ 174.048305] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 174.048352] LustreError: 6476:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 174.048397] LustreError: 6476:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 174.063958] Lustre: Skipped 1 previous similar message [ 174.193823] Lustre: server umount lustre-MDT0001 complete [ 175.783192] Lustre: server umount lustre-OST0000 complete [ 177.141205] Lustre: server umount lustre-OST0001 complete [ 179.693342] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 182.967103] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 198.325306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 198.467043] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 198.481916] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 198.518057] Lustre: lustre-MDT0000: new disk, initializing [ 198.546530] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 198.554915] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 199.927814] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 204.965703] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 205.039413] Lustre: 14938:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 205.219087] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 206.789177] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 210.590652] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 212.360222] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 215.395841] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 215.545099] Lustre: lustre-OST0000: new disk, initializing [ 215.547975] Lustre: Skipped 1 previous similar message [ 215.551354] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 215.555073] Lustre: Skipped 2 previous similar messages [ 215.598395] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 215.603930] Lustre: Skipped 1 previous similar message [ 217.038295] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 217.045238] Lustre: Skipped 1 previous similar message [ 217.050160] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 217.102727] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 218.012746] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 221.857628] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 223.134441] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 223.209826] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 224.638737] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 224.768717] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 227.298124] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 227.301088] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 227.304714] LustreError: Skipped 1 previous similar message [ 227.313624] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 227.320045] Lustre: Skipped 1 previous similar message [ 231.852658] Lustre: server umount lustre-OST0000 complete [ 236.000582] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 236.005060] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 236.012154] Lustre: Skipped 1 previous similar message [ 247.263177] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 247.349472] Lustre: server umount lustre-MDT0000 complete [ 247.776591] LustreError: 15724:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 247.783063] LustreError: 15724:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 249.073250] LustreError: 14930:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772167522 with bad export cookie 8254837337354198813 [ 249.079799] LustreError: 14930:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 249.079920] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 249.216793] Lustre: server umount lustre-MDT0001 complete [ 250.880260] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 23:45:24 (1772167524) [ 251.842743] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 252.758780] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 23:45:26 (1772167526) [ 255.913612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 256.117565] LustreError: 18939:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 256.159370] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 257.519998] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 261.212066] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 262.878190] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 265.425381] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 266.821412] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 269.839638] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 271.937606] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 274.421892] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 290.788983] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 290.790979] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 290.796946] Lustre: Skipped 2 previous similar messages [ 290.800932] Lustre: Skipped 5 previous similar messages [ 290.807830] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 294.117468] Lustre: server umount lustre-OST0000 complete [ 310.239103] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 310.296451] Lustre: server umount lustre-MDT0000 complete [ 311.753327] LustreError: 19656:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772167585 with bad export cookie 8254837337354199695 [ 311.757365] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 311.758992] LustreError: 19656:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 311.932241] Lustre: server umount lustre-MDT0001 complete [ 315.989864] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 317.521623] Key type lgssc unregistered [ 317.691540] LNet: 22138:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 317.696116] LNetError: 22138:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 317.709520] LNet: Removed LNI 192.168.204.154@tcp [ 318.155121] Key type .llcrypt unregistered [ 318.156683] Key type ._llcrypt unregistered [ 328.683096] Key type ._llcrypt registered [ 328.684564] Key type .llcrypt registered [ 328.731365] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 329.059990] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 329.096327] alg: No test for adler32 (adler32-zlib) [ 329.956332] Lustre: Lustre: Build Version: 2.17.50_152_g82f9dcf [ 330.035210] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 331.615151] Key type lgssc registered [ 332.195191] Lustre: Echo OBD driver; http://www.lustre.org/ [ 335.828319] loop: module loaded [ 335.854731] loop0: detected capacity change from 0 to 204800000 [ 335.968925] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 335.974333] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 336.110191] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 338.536099] loop0: detected capacity change from 0 to 204800000 [ 338.589781] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 338.595837] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 338.599470] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 340.754170] loop0: detected capacity change from 0 to 204800000 [ 343.274213] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 343.284694] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 344.404611] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 344.426127] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 344.468231] Lustre: lustre-MDT0000: new disk, initializing [ 344.493134] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 344.500659] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 345.992866] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 350.355909] loop1: detected capacity change from 0 to 204800000 [ 352.456659] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 352.563077] Lustre: lustre-OST07c6: new disk, initializing [ 352.566041] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 352.567089] Lustre: lustre-OST07c6: Not available for connect from 0@lo (not set up) [ 352.569226] Lustre: Skipped 1 previous similar message [ 352.594174] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 354.882632] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 357.869979] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 357.874665] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 357.887215] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 362.741836] Lustre: 24272:0:(osd_internal.h:1455:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 362.748514] Lustre: 24272:0:(osd_handler.c:2093:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 362.753409] Lustre: 24272:0:(osd_handler.c:2100:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 362.757843] Lustre: 24272:0:(osd_handler.c:2110:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 362.762629] Lustre: 24272:0:(osd_handler.c:2117:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 362.768100] Lustre: 24272:0:(osd_handler.c:2124:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 377.212253] Lustre: 24273:0:(osd_internal.h:1455:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 377.218719] Lustre: 24273:0:(osd_handler.c:2093:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 377.223080] Lustre: 24273:0:(osd_handler.c:2100:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 377.227931] Lustre: 24273:0:(osd_handler.c:2110:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/4 [ 377.233088] Lustre: 24273:0:(osd_handler.c:2117:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 377.237406] Lustre: 24273:0:(osd_handler.c:2124:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 381.019179] Lustre: 24273:0:(osd_internal.h:1455:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 381.023373] Lustre: 24273:0:(osd_internal.h:1455:osd_trans_exec_op()) Skipped 1 previous similar message [ 381.026128] Lustre: 24273:0:(osd_handler.c:2093:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 381.028486] Lustre: 24273:0:(osd_handler.c:2093:osd_trans_dump_creds()) Skipped 1 previous similar message [ 381.030782] Lustre: 24273:0:(osd_handler.c:2100:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 381.033365] Lustre: 24273:0:(osd_handler.c:2100:osd_trans_dump_creds()) Skipped 1 previous similar message [ 381.035855] Lustre: 24273:0:(osd_handler.c:2110:osd_trans_dump_creds()) write: 2/23/0, punch: 0/0/0, quota 4/150/0 [ 381.038457] Lustre: 24273:0:(osd_handler.c:2110:osd_trans_dump_creds()) Skipped 1 previous similar message [ 381.040865] Lustre: 24273:0:(osd_handler.c:2117:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 381.042980] Lustre: 24273:0:(osd_handler.c:2117:osd_trans_dump_creds()) Skipped 1 previous similar message [ 381.045490] Lustre: 24273:0:(osd_handler.c:2124:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 381.048100] Lustre: 24273:0:(osd_handler.c:2124:osd_trans_dump_creds()) Skipped 1 previous similar message [ 384.105427] Lustre: 24273:0:(osd_internal.h:1455:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 384.109582] Lustre: 24273:0:(osd_internal.h:1455:osd_trans_exec_op()) Skipped 1 previous similar message [ 384.112879] Lustre: 24273:0:(osd_handler.c:2093:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 384.115848] Lustre: 24273:0:(osd_handler.c:2093:osd_trans_dump_creds()) Skipped 1 previous similar message [ 384.118801] Lustre: 24273:0:(osd_handler.c:2100:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 384.122177] Lustre: 24273:0:(osd_handler.c:2100:osd_trans_dump_creds()) Skipped 1 previous similar message [ 384.125252] Lustre: 24273:0:(osd_handler.c:2110:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 384.128487] Lustre: 24273:0:(osd_handler.c:2110:osd_trans_dump_creds()) Skipped 1 previous similar message [ 384.131954] Lustre: 24273:0:(osd_handler.c:2117:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 384.135420] Lustre: 24273:0:(osd_handler.c:2117:osd_trans_dump_creds()) Skipped 1 previous similar message [ 384.138993] Lustre: 24273:0:(osd_handler.c:2124:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 384.142537] Lustre: 24273:0:(osd_handler.c:2124:osd_trans_dump_creds()) Skipped 1 previous similar message [ 390.931702] Lustre: 24272:0:(osd_internal.h:1455:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 390.936758] Lustre: 24272:0:(osd_internal.h:1455:osd_trans_exec_op()) Skipped 3 previous similar messages [ 390.940814] Lustre: 24272:0:(osd_handler.c:2093:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 390.944899] Lustre: 24272:0:(osd_handler.c:2093:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 390.948790] Lustre: 24272:0:(osd_handler.c:2100:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 390.952707] Lustre: 24272:0:(osd_handler.c:2100:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 390.956443] Lustre: 24272:0:(osd_handler.c:2110:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 390.960266] Lustre: 24272:0:(osd_handler.c:2110:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 390.963581] Lustre: 24272:0:(osd_handler.c:2117:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 390.966824] Lustre: 24272:0:(osd_handler.c:2117:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 390.970255] Lustre: 24272:0:(osd_handler.c:2124:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 390.973674] Lustre: 24272:0:(osd_handler.c:2124:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 400.672461] Lustre: 24274:0:(osd_internal.h:1455:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 400.675200] Lustre: 24274:0:(osd_internal.h:1455:osd_trans_exec_op()) Skipped 5 previous similar messages [ 400.677344] Lustre: 24274:0:(osd_handler.c:2093:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 400.679328] Lustre: 24274:0:(osd_handler.c:2093:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 400.681423] Lustre: 24274:0:(osd_handler.c:2100:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 400.683567] Lustre: 24274:0:(osd_handler.c:2100:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 400.685665] Lustre: 24274:0:(osd_handler.c:2110:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 400.687848] Lustre: 24274:0:(osd_handler.c:2110:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 400.689976] Lustre: 24274:0:(osd_handler.c:2117:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 400.692012] Lustre: 24274:0:(osd_handler.c:2117:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 400.694009] Lustre: 24274:0:(osd_handler.c:2124:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 400.696006] Lustre: 24274:0:(osd_handler.c:2124:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 444.899702] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 444.899714] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 444.909055] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 448.174428] Lustre: server umount lustre-OST07c6 complete [ 452.236601] Lustre: server umount lustre-MDT0000 complete [ 457.838236] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 458.824719] Key type lgssc unregistered [ 458.962303] LNet: 27812:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 458.964715] LNetError: 27812:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 458.975273] LNet: Removed LNI 192.168.204.154@tcp [ 459.318973] Key type .llcrypt unregistered [ 459.320740] Key type ._llcrypt unregistered [ 467.753493] Key type ._llcrypt registered [ 467.754923] Key type .llcrypt registered [ 467.793223] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 468.105455] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 468.140576] alg: No test for adler32 (adler32-zlib) [ 468.988134] Lustre: Lustre: Build Version: 2.17.50_152_g82f9dcf [ 469.069448] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 470.647099] Key type lgssc registered [ 470.970074] Lustre: Echo OBD driver; http://www.lustre.org/ [ 473.809903] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 473.817576] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 475.022924] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 476.629332] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 480.436220] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 480.609755] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 481.863469] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 484.207456] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 485.856210] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 488.877478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 489.039351] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 491.434430] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 493.780345] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 499.307404] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 504.805051] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 504.806516] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 504.807349] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 504.813875] Lustre: Skipped 1 previous similar message [ 509.923611] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 509.926963] Lustre: Skipped 2 previous similar messages [ 510.178964] Lustre: server umount lustre-OST0000 complete [ 515.040789] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 515.041376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 515.043845] Lustre: Skipped 1 previous similar message [ 520.160854] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 520.165615] Lustre: Skipped 2 previous similar messages [ 525.280498] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 525.282047] Lustre: Skipped 1 previous similar message [ 526.303239] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 526.381617] Lustre: server umount lustre-MDT0000 complete [ 527.874398] LustreError: 29022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772167801 with bad export cookie 11683568998711518869 [ 527.882065] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 528.041727] Lustre: server umount lustre-MDT0001 complete [ 532.940218] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 534.160178] Key type lgssc unregistered [ 534.321832] LNet: 32424:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 534.326592] LNetError: 32424:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 534.340820] LNet: Removed LNI 192.168.204.154@tcp [ 534.775920] Key type .llcrypt unregistered [ 534.778297] Key type ._llcrypt unregistered [ 544.111290] Key type ._llcrypt registered [ 544.112779] Key type .llcrypt registered [ 544.172676] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 544.497877] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 544.529803] alg: No test for adler32 (adler32-zlib) [ 545.404482] Lustre: Lustre: Build Version: 2.17.50_152_g82f9dcf [ 545.529355] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 547.143182] Key type lgssc registered [ 547.736443] Lustre: Echo OBD driver; http://www.lustre.org/ [ 550.691465] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 550.700842] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 551.907674] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 553.222105] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 556.786252] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 557.004954] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 558.368435] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 560.577126] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 562.220060] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 565.067686] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 565.190226] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 567.339499] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 569.875139] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 576.495287] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 586.724652] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 586.725236] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 586.725690] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 586.729601] Lustre: Skipped 1 previous similar message [ 586.732488] LustreError: Skipped 1 previous similar message [ 590.307069] Lustre: server umount lustre-OST0000 complete [ 591.841442] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 591.842563] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 591.849514] Lustre: Skipped 1 previous similar message [ 591.853757] Lustre: Skipped 1 previous similar message [ 596.961508] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 596.965672] Lustre: Skipped 1 previous similar message [ 597.877702] Lustre: server umount lustre-MDT0000 complete [ 599.705384] LustreError: 33633:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772167873 with bad export cookie 17386904679239191551 [ 599.710252] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 599.828597] Lustre: server umount lustre-MDT0001 complete [ 604.704790] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 606.204531] Key type lgssc unregistered [ 606.358501] LNet: 36850:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 606.361639] LNetError: 36850:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 606.369377] LNet: Removed LNI 192.168.204.154@tcp [ 606.800237] Key type .llcrypt unregistered [ 606.802369] Key type ._llcrypt unregistered [ 617.138510] Key type ._llcrypt registered [ 617.139983] Key type .llcrypt registered [ 617.183300] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 617.510812] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 617.565482] alg: No test for adler32 (adler32-zlib) [ 618.437645] Lustre: Lustre: Build Version: 2.17.50_152_g82f9dcf [ 618.525761] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 620.119207] Key type lgssc registered [ 620.662826] Lustre: Echo OBD driver; http://www.lustre.org/ [ 624.287994] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 624.299853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 625.549703] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 627.279046] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 631.040977] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 631.243665] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 632.953887] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 635.132299] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 636.521738] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 639.638113] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 639.795027] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 642.313670] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 644.857383] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 651.054737] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 660.651648] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.54@tcp (stopping) [ 661.476264] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 661.478643] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 661.484592] Lustre: Skipped 2 previous similar messages [ 661.491644] Lustre: Skipped 2 previous similar messages [ 664.247103] Lustre: server umount lustre-MDT0000 complete [ 665.765802] LustreError: 38896:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 665.777405] LustreError: 38896:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 665.997887] LustreError: 38207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772167939 with bad export cookie 4738000508872464475 [ 666.004220] LustreError: 38207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 666.004252] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 666.592674] LustreError: 38223:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 666.593082] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 666.609920] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 670.885614] LustreError: 38223:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 670.897074] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.54@tcp (stopping) [ 672.210750] Lustre: server umount lustre-MDT0001 complete [ 690.736007] Lustre: server umount lustre-OST0000 complete [ 698.065758] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 699.485861] Key type lgssc unregistered [ 699.685741] LNet: 41753:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 699.691427] LNetError: 41753:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 699.705681] LNet: Removed LNI 192.168.204.154@tcp [ 700.194737] Key type .llcrypt unregistered [ 700.197048] Key type ._llcrypt unregistered [ 711.001280] Key type ._llcrypt registered [ 711.002527] Key type .llcrypt registered [ 711.039781] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 711.348552] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 711.355208] alg: No test for adler32 (adler32-zlib) [ 712.193541] Lustre: Lustre: Build Version: 2.17.50_152_g82f9dcf [ 712.265649] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 713.839199] Key type lgssc registered [ 714.204693] Lustre: Echo OBD driver; http://www.lustre.org/ [ 717.391707] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 717.401049] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 718.567860] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 719.871493] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 723.074476] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 723.186321] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 724.423123] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 726.640507] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 727.853500] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 730.598085] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 730.747739] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 732.876394] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 734.846895] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 744.112122] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 749.538062] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 749.538649] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 749.540268] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 749.546118] Lustre: Skipped 1 previous similar message [ 751.798515] Lustre: lustre-OST0000: Not available for connect from 192.168.204.54@tcp (stopping) [ 751.803110] Lustre: Skipped 1 previous similar message [ 754.658659] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 754.663282] Lustre: Skipped 1 previous similar message [ 755.352900] Lustre: server umount lustre-OST0000 complete [ 768.557419] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 768.561673] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 768.563118] Lustre: Skipped 1 previous similar message [ 773.600442] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 773.602075] Lustre: Skipped 1 previous similar message [ 774.817442] Lustre: server umount lustre-MDT0000 complete [ 776.254203] LustreError: 43110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772168050 with bad export cookie 13566871041370127397 [ 776.258729] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 776.260993] LustreError: 43110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 782.506327] Lustre: server umount lustre-MDT0001 complete [ 787.531730] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 789.129333] Key type lgssc unregistered [ 789.311101] LNet: 46500:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 789.314755] LNetError: 46500:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 789.324648] LNet: Removed LNI 192.168.204.154@tcp [ 789.809851] Key type .llcrypt unregistered [ 789.812626] Key type ._llcrypt unregistered [ 800.724024] Key type ._llcrypt registered [ 800.725875] Key type .llcrypt registered [ 800.779973] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 801.082419] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 801.089690] alg: No test for adler32 (adler32-zlib) [ 802.005939] Lustre: Lustre: Build Version: 2.17.50_152_g82f9dcf [ 802.115982] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 803.727181] Key type lgssc registered [ 804.313312] Lustre: Echo OBD driver; http://www.lustre.org/ [ 807.780265] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 807.792870] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 809.020481] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 810.812461] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 814.777739] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 814.927553] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 816.580633] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 819.370728] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 820.774412] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 823.821402] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 824.023542] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 826.612531] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 828.930919] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 833.260630] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 836.597987] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 838.376139] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 849.336353] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-02-2623h55m22s [ 850.119896] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 853.987122] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 853.989975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 853.994713] Lustre: Skipped 1 previous similar message [ 854.001228] Lustre: Skipped 2 previous similar messages [ 854.704265] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.54@tcp (stopping) [ 855.716274] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.54@tcp (stopping) [ 855.719145] Lustre: Skipped 3 previous similar messages [ 860.773086] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.54@tcp (stopping) [ 865.247161] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 865.310796] Lustre: server umount lustre-MDT0000 complete [ 866.273952] LustreError: 49490:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 866.280429] LustreError: 49490:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 866.521206] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 866.980834] LustreError: 47873:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 869.160447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 869.218505] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 869.311353] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 869.343838] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 870.375465] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 870.710197] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 871.940184] Lustre: DEBUG MARKER: Wait for df (35981) ... [ 872.456876] Lustre: DEBUG MARKER: done [ 874.465566] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 874.468036] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 874.471569] Lustre: Skipped 1 previous similar message [ 874.473390] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 874.476985] Lustre: Skipped 3 previous similar messages [ 875.500257] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 875.743484] Lustre: server umount lustre-OST0000 complete [ 880.608581] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 880.608721] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 880.614458] Lustre: Skipped 1 previous similar message [ 883.507906] Lustre: server umount lustre-MDT0000 complete [ 885.149282] LustreError: 47901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772168158 with bad export cookie 450706697686594940 [ 885.156535] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 885.158118] LustreError: 47901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 885.327427] Lustre: server umount lustre-MDT0001 complete [ 889.999900] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 891.316279] Key type lgssc unregistered [ 891.460764] LNet: 52424:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 891.465959] LNetError: 52424:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 891.481509] LNet: Removed LNI 192.168.204.154@tcp [ 891.881660] Key type .llcrypt unregistered [ 891.883883] Key type ._llcrypt unregistered [ 910.069350] Key type ._llcrypt registered [ 910.070435] Key type .llcrypt registered [ 910.109423] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 910.601487] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 910.669945] alg: No test for adler32 (adler32-zlib) [ 911.554019] Lustre: Lustre: Build Version: 2.17.50_152_g82f9dcf [ 911.659667] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 913.255225] Key type lgssc registered [ 913.957703] Lustre: Echo OBD driver; http://www.lustre.org/ [ 917.648799] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 917.658076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 918.778153] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 918.790423] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 918.856697] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 920.236957] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 923.568103] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 923.622390] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 923.639963] Lustre: 54592:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 923.778850] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 925.090541] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 927.256892] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 928.660373] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 931.809046] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 931.925443] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 931.997923] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 932.725628] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 934.216173] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 936.293102] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 940.101578] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-02-2623h56m53s [ 940.782693] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 943.839780] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 944.796437] Lustre: DEBUG MARKER: Creating a test file and stat it [ 944.910059] LustreError: 54599:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 965.304836] Lustre: lustre-MDT0000: Client 477c8c97-70b4-45e9-9bb8-fcd6c824ec1a (at 192.168.204.54@tcp) reconnecting [ 984.951131] LustreError: 54599:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 986.001191] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 987.035273] Lustre: DEBUG MARKER: done [ 989.154291] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 989.154486] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 989.159002] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 989.160386] LustreError: Skipped 1 previous similar message [ 989.163752] Lustre: Skipped 1 previous similar message [ 994.273404] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 994.274928] Lustre: Skipped 1 previous similar message [ 994.834809] Lustre: server umount lustre-OST0000 complete [ 999.394215] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 999.395843] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 999.401699] Lustre: Skipped 2 previous similar messages [ 999.408200] Lustre: Skipped 1 previous similar message [ 1002.143863] Lustre: server umount lustre-MDT0000 complete [ 1003.338246] LustreError: 54585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772168277 with bad export cookie 13945909594876733082 [ 1003.341783] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1003.343908] LustreError: 54585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1003.457405] Lustre: server umount lustre-MDT0001 complete [ 1007.568913] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 1008.610219] Key type lgssc unregistered [ 1008.762894] LNet: 58559:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1008.768172] LNetError: 58559:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1008.777733] LNet: Removed LNI 192.168.204.154@tcp [ 1009.279737] Key type .llcrypt unregistered [ 1009.281958] Key type ._llcrypt unregistered [ 1029.570823] Key type ._llcrypt registered [ 1029.571798] Key type .llcrypt registered [ 1029.606152] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 1030.041062] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1030.049982] alg: No test for adler32 (adler32-zlib) [ 1030.908047] Lustre: Lustre: Build Version: 2.17.50_152_g82f9dcf [ 1030.992972] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 1032.575102] Key type lgssc registered [ 1032.925413] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1034.025538] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1034.035196] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1035.153142] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1035.161301] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1035.236669] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1035.661679] Lustre: Failing over lustre-MDT0000 [ 1035.788642] Lustre: server umount lustre-MDT0000 complete [ 1038.456235] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 23:58:32 (1772168312) [ 1041.217737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1041.472847] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1043.271140] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1046.682567] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1046.716798] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1046.726811] Lustre: 61319:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1046.819040] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1048.598499] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1050.597231] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1052.356202] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1055.448785] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1055.570238] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1055.666080] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1057.858848] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1060.274207] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1065.018690] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1069.028079] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1072.608540] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1072.613697] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1072.622734] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1075.170376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1075.174213] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1075.176247] Lustre: Skipped 1 previous similar message [ 1075.184638] Lustre: Skipped 1 previous similar message [ 1077.463906] Lustre: server umount lustre-MDT0000 complete [ 1078.928369] LustreError: 62037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772168352 with bad export cookie 11064113343760304170 [ 1078.931844] LustreError: 62037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1078.932872] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1079.065516] Lustre: server umount lustre-MDT0001 complete [ 1080.516406] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1082.500142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1086.300137] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1088.415221] LustreError: 64732:0:(import.c:337:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 1088.415895] LustreError: 60241:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000007584ae3a x1858253151569024/t0(0) o250->MGC192.168.204.154@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1088.438996] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x998ba077497e1106 [ 1088.444723] Lustre: MGC192.168.204.154@tcp: Connection restored to 0@lo (at 0@lo) [ 1088.666805] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1088.701926] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1090.270288] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1093.553567] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1093.685488] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1095.366565] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1095.784461] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1096.799099] Lustre: 60243:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772168354/real 1772168354] req@00000000c75816db x1858253151568640/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1772168370 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1097.593448] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1099.125685] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1099.872122] Lustre: 60243:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772168357/real 1772168357] req@0000000063c77236 x1858253151568768/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1772168373 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1101.281008] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1101.282496] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1101.284448] Lustre: Skipped 2 previous similar messages [ 1101.287589] Lustre: Skipped 3 previous similar messages [ 1102.303168] Lustre: 60244:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772168359/real 1772168359] req@0000000036b77280 x1858253151568896/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1772168375 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1104.863140] Lustre: 60245:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772168362/real 1772168362] req@000000006367592e x1858253151569152/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1772168378 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1106.401909] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1107.184726] Lustre: server umount lustre-MDT0000 complete [ 1108.833271] LustreError: 61351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772168382 with bad export cookie 11064113343760306438 [ 1108.839782] LustreError: 61351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1108.841100] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1108.981466] Lustre: server umount lustre-MDT0001 complete [ 1112.144396] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1116.663445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1118.688485] LustreError: 60241:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000004a6298cf x1858253151588480/t0(0) o250->MGC192.168.204.154@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1118.700320] LustreError: 60241:0:(client.c:1390:ptlrpc_import_delay_req()) Skipped 2 previous similar messages [ 1118.876846] LustreError: 66151:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1118.888112] LustreError: 66151:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1118.918546] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1118.920763] Lustre: Skipped 1 previous similar message [ 1118.963149] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1120.713325] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1124.226904] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1124.323745] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1124.326132] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1124.327407] Lustre: Skipped 2 previous similar messages [ 1126.135736] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1127.455140] Lustre: 60243:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772168385/real 1772168385] req@00000000505b28ac x1858253151588096/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1772168401 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1128.347545] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1129.450687] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1129.453444] Lustre: Skipped 1 previous similar message [ 1129.697105] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1131.504164] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1134.561821] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1134.566823] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1134.575472] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1135.583139] Lustre: 60242:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772168392/real 1772168392] req@000000005fde68a8 x1858253151588608/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1772168408 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1135.592462] Lustre: 60242:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 1138.183609] Lustre: server umount lustre-MDT0000 complete [ 1139.731937] LustreError: 61351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772168413 with bad export cookie 11064113343760307817 [ 1139.738239] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1139.867425] Lustre: server umount lustre-MDT0001 complete [ 1142.449195] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1143.561051] Lustre: DEBUG MARKER: files compared the same [ 1154.531758] Lustre: server umount lustre-OST0000 complete [ 1160.343073] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 1161.676943] Key type lgssc unregistered [ 1161.878767] LNet: 70236:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1161.883483] LNetError: 70236:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1161.890626] LNet: Removed LNI 192.168.204.154@tcp [ 1162.351596] Key type .llcrypt unregistered [ 1162.353524] Key type ._llcrypt unregistered [ 1173.109314] Key type ._llcrypt registered [ 1173.110868] Key type .llcrypt registered [ 1173.162235] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 1173.481696] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1173.491290] alg: No test for adler32 (adler32-zlib) [ 1174.402924] Lustre: Lustre: Build Version: 2.17.50_152_g82f9dcf [ 1174.507401] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 1176.103203] Key type lgssc registered [ 1176.748798] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1180.295296] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1180.309798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1181.520633] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1183.016651] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug +malloc all [ 1186.688311] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1186.861344] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1188.505311] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug +malloc all [ 1190.501574] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1192.134370] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1195.211595] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1195.346469] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1197.274518] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug +malloc all [ 1199.624003] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1207.652452] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1212.897290] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1212.898123] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1212.909090] Lustre: Skipped 1 previous similar message [ 1212.915195] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1218.016399] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1218.019776] Lustre: Skipped 2 previous similar messages [ 1223.136442] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1223.138323] Lustre: Skipped 1 previous similar message [ 1227.231184] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1227.305571] Lustre: server umount lustre-OST0000 complete [ 1232.863642] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1232.866468] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1232.870840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1235.263256] Lustre: server umount lustre-MDT0000 complete [ 1237.148421] LustreError: 73787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772168510 with bad export cookie 51622812288204347 [ 1237.150630] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1237.155898] LustreError: 73787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1237.293563] Lustre: server umount lustre-MDT0001 complete [ 1241.894405] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 1243.506133] Key type lgssc unregistered [ 1243.694803] LNet: 74824:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1243.699800] LNetError: 74824:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1243.708823] LNet: Removed LNI 192.168.204.154@tcp [ 1244.152228] Key type .llcrypt unregistered [ 1244.155279] Key type ._llcrypt unregistered [ 1254.928304] Key type ._llcrypt registered [ 1254.929721] Key type .llcrypt registered [ 1254.979524] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 1255.406341] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1255.470201] alg: No test for adler32 (adler32-zlib) [ 1256.334851] Lustre: Lustre: Build Version: 2.17.50_152_g82f9dcf [ 1256.419921] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 1258.007114] Key type lgssc registered [ 1258.460356] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1261.935783] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1261.942991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1277.407499] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1282.527278] LustreError: 76175:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.154@tcp: failed processing log, type 4: rc = -110 [ 1309.151230] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1311.441678] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1313.982956] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1317.459770] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1317.529904] Lustre: *** cfs_fail_loc=706, val=0*** [ 1317.721533] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1317.759041] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 1319.392672] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1322.982323] LustreError: 76201:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1322.988468] LustreError: 76201:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1323.391546] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1323.617819] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1325.489174] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1328.122260] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1329.827283] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1333.216620] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1333.221205] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1333.229093] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1333.729607] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1333.734257] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1333.742609] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1335.275189] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1335.279406] Lustre: Skipped 1 previous similar message [ 1336.999508] Lustre: server umount lustre-OST0000 complete [ 1338.848588] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1338.853578] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1338.862117] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1345.505834] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1345.511275] Lustre: Skipped 3 previous similar messages [ 1353.183127] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1353.269930] Lustre: server umount lustre-MDT0000 complete [ 1354.898338] LustreError: 76181:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772168628 with bad export cookie 11712751241744909794 [ 1354.905886] LustreError: 76181:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1354.908812] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1355.097194] Lustre: server umount lustre-MDT0001 complete [ 1359.892650] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 1361.492165] Key type lgssc unregistered [ 1361.670568] LNet: 79438:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1361.675351] LNetError: 79438:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1361.686641] LNet: Removed LNI 192.168.204.154@tcp [ 1362.179591] Key type .llcrypt unregistered [ 1362.182070] Key type ._llcrypt unregistered [ 1373.312830] Key type ._llcrypt registered [ 1373.313930] Key type .llcrypt registered [ 1373.355297] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 1373.693382] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1373.746715] alg: No test for adler32 (adler32-zlib) [ 1374.609207] Lustre: Lustre: Build Version: 2.17.50_152_g82f9dcf [ 1374.697422] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 1376.279225] Key type lgssc registered [ 1376.734864] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1379.304025] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1379.310211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1381.708768] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1385.328031] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1385.596077] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1387.202199] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1390.868509] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1391.005907] LustreError: 81482:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1391.010525] LustreError: 81482:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1391.017497] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1393.716957] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1396.134892] LustreError: 81486:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1396.145554] LustreError: 81486:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1397.832665] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1397.841769] Lustre: MGS: Client adc0c2ac-5fc7-4452-a0b4-35e01df4553e (at 0@lo) reconnecting [ 1397.860822] LustreError: 81496:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 1397.995719] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1398.047634] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 1399.612128] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1408.486587] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1408.488765] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1408.496020] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1408.502856] Lustre: Skipped 1 previous similar message [ 1413.605925] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1413.609770] Lustre: Skipped 1 previous similar message [ 1414.122498] Lustre: server umount lustre-OST0000 complete [ 1418.724615] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1418.724742] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1418.727567] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1418.732545] Lustre: Skipped 2 previous similar messages [ 1421.981137] Lustre: server umount lustre-MDT0000 complete [ 1423.989694] Lustre: server umount lustre-MDT0001 complete [ 1425.800537] Lustre: server umount MGS complete [ 1430.722197] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 00:05:04 (1772168704) [ 1443.224335] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 1447.383380] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 1463.441850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1464.971002] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1469.956325] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1470.103353] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 1470.134941] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1470.194560] Lustre: lustre-MDT0001: new disk, initializing [ 1470.228495] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1470.241538] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1470.244075] Lustre: 88547:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 1470.247868] LustreError: 88547:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 1471.850858] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1477.033968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1477.185059] Lustre: lustre-OST0000: new disk, initializing [ 1477.188180] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1477.223225] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1479.344162] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1480.095775] Lustre: 89568:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@000000007a03c436 x1858253512279296/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1772168769 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1482.931656] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1485.699155] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1485.707549] Lustre: MGS: Client 9efc1dea-6b3a-44b3-96f8-ea3e666526ce (at 0@lo) reconnecting [ 1485.735092] Lustre: 90226:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1485.751115] LustreError: 88532:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 1485.769836] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1485.816814] Lustre: lustre-MDT0000: new disk, initializing [ 1485.888895] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1485.899313] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1486.944378] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1486.944689] Lustre: 90254:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@000000000a2f98fd x1858253512285312/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1772168776 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1486.959069] Lustre: 90254:0:(client.c:1637:after_reply()) Skipped 1 previous similar message [ 1487.698229] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1488.996745] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1489.001680] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1489.005294] Lustre: Skipped 1 previous similar message [ 1489.015841] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1508.323974] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1508.326387] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1508.331506] Lustre: Skipped 1 previous similar message [ 1508.336565] Lustre: Skipped 2 previous similar messages [ 1512.748831] Lustre: server umount lustre-OST0000 complete [ 1518.562697] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1518.562842] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1518.564111] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1518.580325] Lustre: Skipped 1 previous similar message [ 1520.860057] Lustre: server umount lustre-MDT0000 complete [ 1528.925655] Lustre: server umount lustre-MDT0001 complete [ 1530.753411] Lustre: server umount MGS complete [ 1535.318262] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 00:06:49 (1772168809) [ 1541.928404] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 1543.252391] Key type lgssc unregistered [ 1543.407032] LNet: 92715:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1543.412225] LNetError: 92715:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1543.421386] LNet: Removed LNI 192.168.204.154@tcp [ 1543.828340] Key type .llcrypt unregistered [ 1543.830176] Key type ._llcrypt unregistered [ 1551.860382] Key type ._llcrypt registered [ 1551.861898] Key type .llcrypt registered [ 1551.914391] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 1552.263794] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1552.281480] alg: No test for adler32 (adler32-zlib) [ 1553.191435] Lustre: Lustre: Build Version: 2.17.50_152_g82f9dcf [ 1553.307552] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 1554.911191] Key type lgssc registered [ 1555.427387] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1559.133922] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1559.145614] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1560.437754] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1562.250950] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1565.865824] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1566.026974] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1567.782194] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1570.140929] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1571.552595] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1574.041989] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1574.162311] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1575.981872] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1577.733631] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1585.445441] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:33) [ 1595.875657] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1595.876481] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1595.883733] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1595.886350] Lustre: Skipped 1 previous similar message [ 1600.993217] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1600.996370] Lustre: Skipped 1 previous similar message [ 1605.087132] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1605.136305] Lustre: server umount lustre-OST0000 complete [ 1607.135816] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1607.137690] LustreError: Skipped 1 previous similar message [ 1607.138689] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1607.141291] Lustre: Skipped 1 previous similar message [ 1607.142730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1611.232153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1611.232574] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1612.565606] Lustre: server umount lustre-MDT0000 complete [ 1613.825220] LustreError: 96080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772168887 with bad export cookie 14350671985829058447 [ 1613.829032] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1613.829156] LustreError: 96080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1613.949850] Lustre: server umount lustre-MDT0001 complete [ 1617.580866] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 1618.636723] Key type lgssc unregistered [ 1618.771427] LNet: 97115:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1618.774156] LNetError: 97115:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1618.780343] LNet: Removed LNI 192.168.204.154@tcp [ 1619.101883] Key type .llcrypt unregistered [ 1619.102939] Key type ._llcrypt unregistered [ 1625.322653] Key type ._llcrypt registered [ 1625.323593] Key type .llcrypt registered [ 1625.363258] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 1625.803738] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1625.822690] alg: No test for adler32 (adler32-zlib) [ 1626.682783] Lustre: Lustre: Build Version: 2.17.50_152_g82f9dcf [ 1626.767469] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 1628.343151] Key type lgssc registered [ 1628.689893] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1630.694897] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1630.695802] LustreError: 98159:0:(tgt_mount.c:2535:server_fill_super()) cfs_race id 716 sleeping [ 1630.697814] Lustre: Skipped 1 previous similar message [ 1630.697821] LustreError: 98158:0:(tgt_mount.c:2535:server_fill_super()) cfs_fail_race id 716 waking [ 1630.704412] LustreError: 98159:0:(tgt_mount.c:2535:server_fill_super()) cfs_fail_race id 716 awake: rc=4995 [ 1630.706708] LustreError: 98159:0:(tgt_mount.c:2547:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 1630.709591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1630.710490] LustreError: 98159:0:(super25.c:190:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1631.871818] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1634.289344] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1634.386272] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1635.613491] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1637.135264] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing lsmod [ 1638.119112] LustreError: 99330:0:(tgt_mount.c:2535:server_fill_super()) cfs_race id 716 sleeping [ 1638.119113] LustreError: 99329:0:(tgt_mount.c:2535:server_fill_super()) cfs_fail_race id 716 waking [ 1638.125182] LustreError: 99330:0:(tgt_mount.c:2535:server_fill_super()) cfs_fail_race id 716 awake: rc=5000 [ 1638.127723] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1638.128672] LustreError: 99330:0:(tgt_mount.c:2547:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 1638.134435] LustreError: 99330:0:(super25.c:190:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1638.241820] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1639.391902] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1639.394110] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1639.398502] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1643.489307] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1643.489650] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1643.491030] Lustre: Skipped 1 previous similar message [ 1643.494178] Lustre: Skipped 1 previous similar message [ 1644.513920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1644.517625] Lustre: Skipped 2 previous similar messages [ 1645.270666] Lustre: server umount lustre-MDT0000 complete [ 1646.587140] LustreError: 98174:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772168920 with bad export cookie 10487187289650305910 [ 1646.590283] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1646.591690] LustreError: 98174:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1648.607295] LustreError: 98626:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000007f332876 x1858253776502144/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1648.664038] Lustre: server umount lustre-MDT0001 complete [ 1660.251933] Lustre: server umount lustre-OST0000 complete [ 1663.050916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1663.212716] LustreError: 100397:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1663.220041] LustreError: 100397:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1663.238621] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1664.517479] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1667.485581] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1667.613255] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1668.938995] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1670.805307] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1671.978033] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1674.203596] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1676.253330] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1678.039636] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1688.675863] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:65) [ 1693.152438] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1693.156177] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1693.162620] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1694.177525] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1696.149384] Lustre: server umount lustre-OST0000 complete [ 1698.271811] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1698.274155] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1698.278473] Lustre: Skipped 1 previous similar message [ 1698.280773] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1698.282482] Lustre: Skipped 2 previous similar messages [ 1703.566221] Lustre: server umount lustre-MDT0000 complete [ 1704.415995] LustreError: 100397:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1704.422035] LustreError: 100397:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1704.804130] LustreError: 100377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772168978 with bad export cookie 10487187289650306582 [ 1704.806772] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1704.808029] LustreError: 100377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1704.917278] Lustre: server umount lustre-MDT0001 complete [ 1708.335561] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 1709.366531] Key type lgssc unregistered [ 1709.490424] LNet: 103591:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1709.493763] LNetError: 103591:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1709.501425] LNet: Removed LNI 192.168.204.154@tcp [ 1709.813213] Key type .llcrypt unregistered [ 1709.814928] Key type ._llcrypt unregistered [ 1718.325996] Key type ._llcrypt registered [ 1718.326767] Key type .llcrypt registered [ 1718.360049] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 1718.759919] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1718.776905] alg: No test for adler32 (adler32-zlib) [ 1719.613711] Lustre: Lustre: Build Version: 2.17.50_152_g82f9dcf [ 1719.689444] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 1721.263153] Key type lgssc registered [ 1721.596136] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1724.015988] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1724.020889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1725.151929] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1726.249119] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1728.928465] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1729.048374] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1730.294189] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1732.068969] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1733.261573] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1735.561682] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1735.663292] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1737.369578] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1739.009721] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1749.990308] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:67 to 0x240000400:97) [ 1756.670465] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 1757.468054] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 1759.199982] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1759.202213] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1759.205789] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1760.228498] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1760.229407] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1760.229745] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1764.497062] Lustre: server umount lustre-OST0000 complete [ 1769.951829] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1769.954337] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1769.958660] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1769.960895] Lustre: Skipped 1 previous similar message [ 1771.870372] Lustre: server umount lustre-MDT0000 complete [ 1773.069522] LustreError: 107182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772169046 with bad export cookie 1920956070598738366 [ 1773.072484] LustreError: 107182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1773.072863] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1773.178129] Lustre: server umount lustre-MDT0001 complete [ 1776.899656] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 1777.912322] Key type lgssc unregistered [ 1778.027446] LNet: 108268:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1778.031465] LNetError: 108268:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1778.043324] LNet: Removed LNI 192.168.204.154@tcp [ 1778.356526] Key type .llcrypt unregistered [ 1778.358127] Key type ._llcrypt unregistered [ 1785.447445] Key type ._llcrypt registered [ 1785.448531] Key type .llcrypt registered [ 1785.483635] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 1785.856034] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1785.884227] alg: No test for adler32 (adler32-zlib) [ 1786.729793] Lustre: Lustre: Build Version: 2.17.50_152_g82f9dcf [ 1786.807454] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 1788.383195] Key type lgssc registered [ 1788.737635] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1791.293564] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1791.300265] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1792.482057] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1793.773543] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1796.631530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1796.767454] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1798.143046] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1799.886458] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1801.120987] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1803.431524] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1803.603741] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1805.416151] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1807.126699] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1818.982460] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:99 to 0x240000400:129) [ 1827.296121] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1827.298870] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1827.303440] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1829.349178] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1829.349643] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1829.349860] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1829.355759] Lustre: Skipped 1 previous similar message [ 1831.832690] Lustre: server umount lustre-OST0000 complete [ 1834.465217] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1834.466309] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1834.470266] Lustre: Skipped 1 previous similar message [ 1839.395665] Lustre: server umount lustre-MDT0000 complete [ 1839.584586] LustreError: 110162:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1839.593254] LustreError: 110162:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1840.921563] LustreError: 109431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772169114 with bad export cookie 11922898318398187012 [ 1840.925555] LustreError: 109431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1840.925856] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1841.056989] Lustre: server umount lustre-MDT0001 complete [ 1844.965295] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 1845.945123] Key type lgssc unregistered [ 1846.058332] LNet: 112653:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1846.060527] LNetError: 112653:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1846.067266] LNet: Removed LNI 192.168.204.154@tcp [ 1846.344480] Key type .llcrypt unregistered [ 1846.345657] Key type ._llcrypt unregistered [ 1855.305651] Key type ._llcrypt registered [ 1855.307489] Key type .llcrypt registered [ 1855.358279] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 1855.675263] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1855.711069] alg: No test for adler32 (adler32-zlib) [ 1856.568043] Lustre: Lustre: Build Version: 2.17.50_152_g82f9dcf [ 1856.653070] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 1858.231147] Key type lgssc registered [ 1858.600382] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1861.220474] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1861.228652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1862.397048] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1863.806561] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1866.682513] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1866.807381] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1868.089037] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1869.831255] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1870.946709] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1873.240985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1873.360948] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1875.215074] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1876.947457] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1887.717544] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:131 to 0x240000400:161) [ 1895.012450] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 1896.271912] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 1896.274060] Lustre: Skipped 2 previous similar messages [ 1897.710471] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 1897.712992] Lustre: Skipped 2 previous similar messages [ 1900.554334] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 1900.566974] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 1904.917857] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.204.54@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 1904.921556] Lustre: Skipped 2 previous similar messages [ 1910.778467] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.54@tcp 192.168.0.[2,10]@tcp [ 1912.774446] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.204.54/24@tcp in log lustre-MDT0000 [ 1912.778243] Lustre: Skipped 2 previous similar messages [ 1915.895182] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.54/24@tcp [ 1915.898254] Lustre: Skipped 1 previous similar message [ 1922.528020] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1922.530830] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1922.535421] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1923.552811] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1923.553246] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1923.553620] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1923.559573] Lustre: Skipped 1 previous similar message [ 1926.281764] Lustre: server umount lustre-OST0000 complete [ 1928.159775] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1928.162029] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1928.165874] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1933.588864] Lustre: server umount lustre-MDT0000 complete [ 1933.791946] LustreError: 114740:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1933.796606] LustreError: 114740:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1934.756241] LustreError: 117490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772169208 with bad export cookie 12629598074365753722 [ 1934.759552] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1934.762242] LustreError: 117490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1934.870806] Lustre: server umount lustre-MDT0001 complete [ 1938.212576] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 1939.204352] Key type lgssc unregistered [ 1939.331488] LNet: 118726:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1939.334550] LNetError: 118726:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1939.342453] LNet: Removed LNI 192.168.204.154@tcp [ 1939.626870] Key type .llcrypt unregistered [ 1939.628623] Key type ._llcrypt unregistered [ 1949.768026] Key type ._llcrypt registered [ 1949.769328] Key type .llcrypt registered [ 1949.814556] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 1950.390454] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1950.400807] alg: No test for adler32 (adler32-zlib) [ 1951.273825] Lustre: Lustre: Build Version: 2.17.50_152_g82f9dcf [ 1951.367560] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 1952.951161] Key type lgssc registered [ 1953.361282] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1955.995681] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1956.002900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1957.155369] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 1957.158628] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1957.164928] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.204.54/24@tcp [ 1958.359463] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1961.193656] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1961.308870] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 1961.317833] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.54/24@tcp [ 1961.335624] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1962.553845] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1964.342773] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1965.518581] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1967.683672] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1967.807065] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1969.649125] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1971.440672] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1982.187860] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:169 to 0x240000400:193) [ 1992.674163] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1992.674487] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1992.677243] Lustre: Skipped 1 previous similar message [ 1992.679195] LustreError: Skipped 1 previous similar message [ 1992.680923] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1992.682847] Lustre: Skipped 1 previous similar message [ 1996.310142] Lustre: server umount lustre-OST0000 complete [ 1997.792182] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1997.795706] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2002.399861] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2002.402307] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2002.406270] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2003.609743] Lustre: server umount lustre-MDT0000 complete [ 2004.785488] LustreError: 120228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772169278 with bad export cookie 3091168386852954551 [ 2004.787524] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2004.788584] LustreError: 120228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2004.907098] Lustre: server umount lustre-MDT0001 complete [ 2008.228452] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 2009.225389] Key type lgssc unregistered [ 2009.353287] LNet: 123503:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2009.355286] LNetError: 123503:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2009.361378] LNet: Removed LNI 192.168.204.154@tcp [ 2009.657136] Key type .llcrypt unregistered [ 2009.658089] Key type ._llcrypt unregistered [ 2018.116552] Key type ._llcrypt registered [ 2018.117617] Key type .llcrypt registered [ 2018.160163] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 2023.885782] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 2024.305144] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2024.338128] alg: No test for adler32 (adler32-zlib) [ 2025.202668] Lustre: Lustre: Build Version: 2.17.50_152_g82f9dcf [ 2025.293269] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 2026.879187] Key type lgssc registered [ 2027.246966] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2038.069755] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2038.074717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2039.162546] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2039.172915] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2039.208461] Lustre: lustre-MDT0000: new disk, initializing [ 2039.231968] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2039.238076] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2040.442552] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2044.456783] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2044.495865] Lustre: 126842:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2044.518758] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2044.521168] Lustre: Skipped 1 previous similar message [ 2044.557568] Lustre: lustre-MDT0001: new disk, initializing [ 2044.586401] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2044.596964] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2044.600719] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2045.797456] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2048.826159] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2050.044307] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2052.344699] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2052.450672] Lustre: lustre-OST0000: new disk, initializing [ 2052.453362] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2052.482658] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2054.261865] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2055.926056] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2055.928666] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2055.956959] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2057.212650] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2058.529667] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2058.597609] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2059.786874] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2059.855435] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2061.282218] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2061.284049] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2061.284707] LustreError: Skipped 1 previous similar message [ 2061.291843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2061.294249] Lustre: Skipped 1 previous similar message [ 2062.541478] Lustre: server umount lustre-OST0000 complete [ 2065.376214] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2065.380903] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2065.387537] Lustre: Skipped 1 previous similar message [ 2065.390640] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2066.400303] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2066.400919] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2071.520319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2071.522431] Lustre: Skipped 2 previous similar messages [ 2076.640398] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2076.643278] Lustre: Skipped 1 previous similar message [ 2078.175089] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2078.262293] Lustre: server umount lustre-MDT0000 complete [ 2079.450270] LustreError: 126834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772169353 with bad export cookie 3523064978824833395 [ 2079.454096] LustreError: 126834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2079.454170] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2079.565725] Lustre: server umount lustre-MDT0001 complete [ 2082.234970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2082.406220] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2083.606509] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2086.294418] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2086.408732] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2087.627089] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2089.349228] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2090.569265] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2092.836794] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2094.867663] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2096.643916] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2097.972742] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2108.211055] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 10 sec [ 2109.468540] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2109.539134] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2112.801285] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2112.862435] Lustre: lustre-OST0001: new disk, initializing [ 2112.864235] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2112.891219] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2112.894287] Lustre: Skipped 1 previous similar message [ 2114.639436] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2117.761768] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2118.143689] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2118.148303] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2118.173777] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2118.849618] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 2120.079415] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2120.158536] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2121.298969] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8cbd04998000.ost_server_uuid 50 [ 2121.763242] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8cbd04998000.ost_server_uuid in FULL state after 0 sec [ 2128.356001] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2128.357732] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2128.359021] LustreError: Skipped 1 previous similar message [ 2128.362635] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2128.365825] Lustre: Skipped 1 previous similar message [ 2129.665446] Lustre: server umount lustre-OST0001 complete [ 2132.960027] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2132.960200] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2132.962452] LustreError: Skipped 1 previous similar message [ 2132.965317] Lustre: Skipped 1 previous similar message [ 2133.475605] LustreError: 132847:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2133.479743] LustreError: 132847:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2138.595037] LustreError: 132242:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2138.599008] LustreError: 132242:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2143.717787] LustreError: 132244:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2143.729542] LustreError: 132244:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2145.247061] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2145.294759] Lustre: server umount lustre-OST0000 complete [ 2147.808089] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2147.810650] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2147.814566] Lustre: Skipped 1 previous similar message [ 2147.816425] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2147.818720] Lustre: Skipped 8 previous similar messages [ 2152.798204] Lustre: server umount lustre-MDT0000 complete [ 2153.951935] LustreError: 135167:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2153.958300] LustreError: 135167:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2153.986386] LustreError: 130632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772169427 with bad export cookie 3523064978824834263 [ 2153.989608] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2153.992218] LustreError: 130632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2154.099585] Lustre: server umount lustre-MDT0001 complete [ 2160.083127] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 2161.021490] Key type lgssc unregistered [ 2161.142356] LNet: 136559:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2161.145034] LNetError: 136559:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2161.154306] LNet: Removed LNI 192.168.204.154@tcp [ 2161.425873] Key type .llcrypt unregistered [ 2161.427283] Key type ._llcrypt unregistered [ 2173.862249] Key type ._llcrypt registered [ 2173.863845] Key type .llcrypt registered [ 2173.905237] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 2179.874296] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 2180.305992] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2180.382264] alg: No test for adler32 (adler32-zlib) [ 2181.241731] Lustre: Lustre: Build Version: 2.17.50_152_g82f9dcf [ 2181.333467] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 2182.919119] Key type lgssc registered [ 2183.249028] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2194.054293] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2194.059675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2195.143525] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2195.153428] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2195.195361] Lustre: lustre-MDT0000: new disk, initializing [ 2195.219774] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2195.227188] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2196.434376] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2200.607989] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2200.650161] Lustre: 140650:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2200.671297] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2200.672793] Lustre: Skipped 1 previous similar message [ 2200.706282] Lustre: lustre-MDT0001: new disk, initializing [ 2200.738219] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2200.749411] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2200.751991] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2201.938039] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2205.096056] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2206.334494] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2208.578805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2208.673821] Lustre: lustre-OST0000: new disk, initializing [ 2208.675694] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2208.700718] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2208.909751] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2208.912545] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2208.941982] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2210.417148] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2213.399241] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2215.289891] Lustre: Failing over lustre-OST0000 [ 2215.357907] Lustre: server umount lustre-OST0000 complete [ 2228.119375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2228.229738] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2228.233310] Lustre: Skipped 1 previous similar message [ 2228.237337] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2228.242646] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2229.932127] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2230.034257] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2231.235429] Lustre: Failing over lustre-MDT0000 [ 2231.328672] Lustre: server umount lustre-MDT0000 complete [ 2231.775801] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2231.779703] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2233.314232] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2235.359102] Lustre: 138876:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772169493/real 1772169493] req@000000003ca616e0 x1858254358472704/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1772169509 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2238.432102] LustreError: 140660:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2238.436781] LustreError: 140660:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 2240.164735] LustreError: 140655:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2240.479078] Lustre: 138876:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772169498/real 1772169498] req@00000000aef8e5ec x1858254358474368/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1772169514 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2243.552281] LustreError: 140660:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2243.559726] LustreError: 140660:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2244.107051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2244.143089] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2244.146581] Lustre: 140652:0:(mgc_request.c:1911:mgc_process_log()) MGC192.168.204.154@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 2244.212615] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 2244.230829] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2244.231251] Lustre: lustre-OST0000: Recovery over after 0:15, of 3 clients 3 recovered and 0 were evicted. [ 2244.238996] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2244.267271] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2245.284338] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2245.499919] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2249.703469] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2249.705143] Lustre: Skipped 1 previous similar message [ 2249.709541] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2249.727141] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2254.817239] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2254.819368] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2254.821238] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2254.822404] Lustre: Skipped 2 previous similar messages [ 2256.783323] Lustre: server umount lustre-OST0000 complete [ 2259.938230] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2259.938518] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2259.940965] LustreError: Skipped 1 previous similar message [ 2259.945178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2259.947061] Lustre: Skipped 2 previous similar messages [ 2264.153638] Lustre: server umount lustre-MDT0000 complete [ 2265.056084] LustreError: 141441:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2265.060525] LustreError: 141441:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2265.397261] LustreError: 140641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772169539 with bad export cookie 5020895991735834819 [ 2265.400401] LustreError: 140641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2265.401103] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2265.525647] Lustre: server umount lustre-MDT0001 complete [ 2269.075992] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 2270.138034] Key type lgssc unregistered [ 2270.269511] LNet: 145876:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2270.273059] LNetError: 145876:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2270.283397] LNet: Removed LNI 192.168.204.154@tcp [ 2270.594187] Key type .llcrypt unregistered [ 2270.595677] Key type ._llcrypt unregistered [ 2290.425299] Key type ._llcrypt registered [ 2290.426275] Key type .llcrypt registered [ 2290.466509] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 2290.856610] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2290.892180] alg: No test for adler32 (adler32-zlib) [ 2291.757256] Lustre: Lustre: Build Version: 2.17.50_152_g82f9dcf [ 2291.839205] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 2293.423169] Key type lgssc registered [ 2293.796885] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2296.231995] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2296.240377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2297.388835] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2298.610742] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2301.575926] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2303.530844] LustreError: 147405:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2303.535502] LustreError: 147405:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2303.539336] LustreError: 147405:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 6, retries 0, failed: rc = -5 [ 2303.651397] Lustre: server umount lustre-MDT0000 complete [ 2308.295817] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 2311.259704] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 2322.277622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2322.387748] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2322.399992] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2322.442411] Lustre: lustre-MDT0000: new disk, initializing [ 2322.468735] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2322.475603] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2323.807284] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2327.878981] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2327.919778] Lustre: 150555:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2327.945729] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2327.947159] Lustre: Skipped 1 previous similar message [ 2327.982320] Lustre: lustre-MDT0001: new disk, initializing [ 2328.013674] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2328.024822] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2328.028556] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2329.187042] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2332.115570] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2333.323087] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2335.410705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2335.514778] Lustre: lustre-OST0000: new disk, initializing [ 2335.516565] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2335.541058] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2337.284332] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2340.214042] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2344.439194] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2344.442035] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2344.474191] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2347.326582] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 2347.895212] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 00:20:21 (1772169621) [ 2405.855969] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2405.856453] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2405.861627] Lustre: Skipped 1 previous similar message [ 2410.975562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2410.978056] Lustre: Skipped 4 previous similar messages [ 2411.362655] Lustre: server umount lustre-MDT0000 complete [ 2412.530487] LustreError: 150547:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772169686 with bad export cookie 908764016717385304 [ 2412.531534] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2412.534128] LustreError: 150547:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2416.095487] LustreError: 151342:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2416.095552] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2416.099121] LustreError: 151342:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2416.104645] Lustre: Skipped 1 previous similar message [ 2416.106052] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2417.119420] LustreError: 151342:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2418.727178] Lustre: server umount lustre-MDT0001 complete [ 2426.124293] Lustre: server umount lustre-OST0000 complete [ 2429.065870] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 2432.011027] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 2443.688390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2443.780654] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2443.793592] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2443.838000] Lustre: lustre-MDT0000: new disk, initializing [ 2443.861278] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2443.868263] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2445.121333] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2449.175488] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2449.214338] Lustre: 156777:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2449.235789] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2449.238709] Lustre: Skipped 1 previous similar message [ 2449.275308] Lustre: lustre-MDT0001: new disk, initializing [ 2449.320532] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2449.324858] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2450.567574] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2453.804183] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2454.969869] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2457.245774] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2457.346371] Lustre: lustre-OST0000: new disk, initializing [ 2457.348241] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2457.374737] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2457.376622] Lustre: Skipped 1 previous similar message [ 2458.821363] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2458.826843] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2458.865249] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2459.383733] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2462.358086] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2463.698609] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2463.764093] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2465.048562] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2465.118221] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2469.346049] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2469.346570] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2469.348160] LustreError: Skipped 1 previous similar message [ 2469.352878] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2469.355709] Lustre: Skipped 2 previous similar messages [ 2474.465151] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2474.466963] Lustre: Skipped 1 previous similar message [ 2480.095117] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2480.153969] Lustre: server umount lustre-OST0000 complete [ 2484.704902] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2484.705436] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2484.708392] Lustre: Skipped 2 previous similar messages [ 2484.713042] Lustre: Skipped 3 previous similar messages [ 2487.525234] Lustre: server umount lustre-MDT0000 complete [ 2488.704063] LustreError: 156769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772169762 with bad export cookie 908764016717544120 [ 2488.706834] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2488.706934] LustreError: 156769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2488.824846] Lustre: server umount lustre-MDT0001 complete [ 2491.421905] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 00:22:45 (1772169765) [ 2496.344093] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 2499.073398] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 2511.729794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2511.838969] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2511.850987] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2511.896244] Lustre: lustre-MDT0000: new disk, initializing [ 2511.922627] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2511.928832] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2513.255638] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2517.626590] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2517.676774] Lustre: 163095:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2517.804251] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2519.416937] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2522.694788] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2523.809693] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2526.306501] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2528.528566] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2529.090235] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2529.138973] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2531.800890] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2534.375066] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2534.375737] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2534.378634] LustreError: Skipped 1 previous similar message [ 2534.386123] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2534.390747] Lustre: Skipped 1 previous similar message [ 2539.742275] Lustre: server umount lustre-OST0000 complete [ 2543.584131] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2543.587070] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2543.591840] Lustre: Skipped 1 previous similar message [ 2555.359136] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2555.439891] Lustre: server umount lustre-MDT0000 complete [ 2556.878956] LustreError: 163086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772169830 with bad export cookie 908764016717544988 [ 2556.885226] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2556.885963] LustreError: 163086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2557.040952] Lustre: server umount lustre-MDT0001 complete [ 2559.872064] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 00:23:53 (1772169833) [ 2565.153500] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 2568.514457] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 2580.542820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2580.637278] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2580.649921] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2580.651511] Lustre: Skipped 3 previous similar messages [ 2580.692691] Lustre: lustre-MDT0000: new disk, initializing [ 2580.694222] Lustre: Skipped 2 previous similar messages [ 2580.721457] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2580.723408] Lustre: Skipped 2 previous similar messages [ 2580.729226] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2580.731483] Lustre: Skipped 2 previous similar messages [ 2582.020423] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2586.287153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2586.333535] Lustre: 169012:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2586.449378] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2587.855500] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2590.983956] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2592.203429] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2594.741768] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2595.424935] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2596.807936] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2599.954766] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2605.538039] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2605.538376] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2605.541057] LustreError: Skipped 1 previous similar message [ 2605.544727] Lustre: Skipped 1 previous similar message [ 2605.546737] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2605.550104] Lustre: Skipped 10 previous similar messages [ 2608.096829] Lustre: server umount lustre-OST0000 complete [ 2615.514786] Lustre: server umount lustre-MDT0000 complete [ 2615.775903] LustreError: 169024:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2615.781833] LustreError: 169024:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2616.692449] LustreError: 169796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772169890 with bad export cookie 908764016717546059 [ 2616.695380] LustreError: 169796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2616.696039] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2616.830942] Lustre: server umount lustre-MDT0001 complete [ 2620.159490] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 2621.208819] Key type lgssc unregistered [ 2621.325430] LNet: 172490:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2621.328693] LNetError: 172490:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2621.340377] LNet: Removed LNI 192.168.204.154@tcp [ 2621.625920] Key type .llcrypt unregistered [ 2621.626957] Key type ._llcrypt unregistered [ 2630.586544] Key type ._llcrypt registered [ 2630.587536] Key type .llcrypt registered [ 2630.627528] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 2631.123988] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2631.131759] alg: No test for adler32 (adler32-zlib) [ 2631.993145] Lustre: Lustre: Build Version: 2.17.50_152_g82f9dcf [ 2632.081952] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 2633.671169] Key type lgssc registered [ 2634.159663] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2637.111658] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2637.118555] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2638.288361] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2639.534157] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2642.480630] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2642.595164] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2643.721825] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2645.561538] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2646.922882] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2649.200023] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2649.316517] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2651.213028] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2652.801404] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2663.653428] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2683.360305] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2683.363616] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2683.369808] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2684.385686] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2684.387113] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2684.387575] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2684.392747] Lustre: Skipped 1 previous similar message [ 2686.226922] Lustre: server umount lustre-OST0000 complete [ 2688.992368] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2688.997612] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2689.007323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2693.744626] Lustre: server umount lustre-MDT0000 complete [ 2694.624244] LustreError: 173865:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2694.629119] LustreError: 173865:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2695.047317] LustreError: 174582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772169968 with bad export cookie 6016301416758560792 [ 2695.050379] LustreError: 174582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2695.050789] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2695.175505] Lustre: server umount lustre-MDT0001 complete [ 2699.931460] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 2701.507790] Key type lgssc unregistered [ 2701.681696] LNet: 177064:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2701.686778] LNetError: 177064:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2701.695686] LNet: Removed LNI 192.168.204.154@tcp [ 2702.158628] Key type .llcrypt unregistered [ 2702.161070] Key type ._llcrypt unregistered [ 2712.422147] Key type ._llcrypt registered [ 2712.423169] Key type .llcrypt registered [ 2712.460686] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 2712.828660] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2712.859831] alg: No test for adler32 (adler32-zlib) [ 2713.702318] Lustre: Lustre: Build Version: 2.17.50_152_g82f9dcf [ 2713.789215] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 2715.367260] Key type lgssc registered [ 2715.923586] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2718.438261] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2718.446049] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2719.606483] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2721.123094] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2724.435210] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2724.560818] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2725.754424] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2727.737205] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2729.275461] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2732.236687] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2732.350264] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2734.697730] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2736.720306] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2744.684152] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 2749.922780] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2749.924270] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2749.927475] LustreError: Skipped 1 previous similar message [ 2749.936075] Lustre: Skipped 1 previous similar message [ 2749.939227] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2749.945677] Lustre: Skipped 1 previous similar message [ 2752.174249] Lustre: lustre-OST0000: Not available for connect from 192.168.204.54@tcp (stopping) [ 2755.041703] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2755.044481] Lustre: Skipped 1 previous similar message [ 2755.803811] Lustre: server umount lustre-OST0000 complete [ 2758.158798] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8cbd051d3800.ost_server_uuid 50 [ 2758.841890] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 2770.400944] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2770.401639] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2770.405270] Lustre: Skipped 1 previous similar message [ 2775.414302] Lustre: server umount lustre-MDT0000 complete [ 2775.521726] LustreError: 179150:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2775.534455] LustreError: 179150:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2777.156353] LustreError: 178420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772170050 with bad export cookie 2938202566012020148 [ 2777.160389] LustreError: 178420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2777.160535] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2777.275903] Lustre: server umount lustre-MDT0001 complete [ 2780.984782] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 00:27:34 (1772170054) [ 2784.343513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2784.532675] LustreError: 181921:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2784.567427] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2785.977797] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2789.857171] LustreError: 181921:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2790.047068] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2790.169673] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2791.676805] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2793.876898] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2795.534436] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2798.385143] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2798.499158] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2800.457778] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2802.442679] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2805.194638] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2805.307476] Lustre: lustre-OST0001: new disk, initializing [ 2805.310839] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2805.957325] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2805.963541] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2806.013473] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2807.680214] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2810.088460] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 2811.020792] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2814.944944] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2814.949761] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2814.961112] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2814.965122] Lustre: Skipped 1 previous similar message [ 2818.311895] Lustre: server umount lustre-OST0000 complete [ 2820.578074] LustreError: 184519:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2820.582545] LustreError: 184519:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2820.837768] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2820.921044] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 2822.412200] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2822.481619] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 2825.698096] LustreError: 185386:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2825.705705] LustreError: 185386:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2835.940829] LustreError: 183513:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2835.946306] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2835.947479] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2835.947487] Lustre: Skipped 3 previous similar messages [ 2835.950983] LustreError: 183513:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2835.958820] Lustre: Skipped 2 previous similar messages [ 2838.770172] Lustre: server umount lustre-OST0001 complete [ 2841.058235] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2841.066324] Lustre: Skipped 1 previous similar message [ 2846.839035] Lustre: server umount lustre-MDT0000 complete [ 2848.602054] LustreError: 181901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772170122 with bad export cookie 2938202566012021205 [ 2848.608984] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2848.612214] LustreError: 181901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2848.763151] Lustre: server umount lustre-MDT0001 complete [ 2858.581866] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 00:28:52 (1772170132) [ 2861.256743] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2861.415755] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2861.436700] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2861.544445] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2861.548590] Lustre: Skipped 1 previous similar message [ 2862.930974] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2867.025784] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2867.088951] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2867.114313] Lustre: 187692:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2869.057674] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2871.506697] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2873.301288] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2876.460057] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2876.525995] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2878.833682] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2881.417995] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2884.294605] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2884.327929] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2884.484244] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 2884.494750] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 2886.444137] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2889.046846] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2894.516765] Lustre: lustre-OST0000: Not available for connect from 192.168.204.54@tcp (stopping) [ 2894.521760] Lustre: Skipped 5 previous similar messages [ 2894.819842] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2894.820260] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2894.825004] LustreError: Skipped 2 previous similar messages [ 2896.453329] Lustre: server umount lustre-OST0000 complete [ 2899.622320] LustreError: 189313:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2910.184444] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2910.189545] LustreError: Skipped 1 previous similar message [ 2913.892350] Lustre: server umount lustre-OST0001 complete [ 2918.367971] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2918.370241] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2918.373281] Lustre: Skipped 3 previous similar messages [ 2921.842785] Lustre: server umount lustre-MDT0000 complete [ 2923.464545] LustreError: 187683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772170197 with bad export cookie 2938202566012022563 [ 2923.468042] LustreError: 187683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2923.469179] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2923.576772] Lustre: server umount lustre-MDT0001 complete [ 2933.536346] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 00:30:07 (1772170207) [ 2940.808453] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 2944.954798] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 2959.790040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2959.941318] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2959.954958] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2960.006977] Lustre: lustre-MDT0000: new disk, initializing [ 2960.045460] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2960.048927] Lustre: Skipped 3 previous similar messages [ 2960.057876] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2961.666118] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2967.325836] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2967.388728] Lustre: 195264:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2967.422869] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2967.426235] Lustre: Skipped 1 previous similar message [ 2967.469433] Lustre: lustre-MDT0001: new disk, initializing [ 2967.521624] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2967.525431] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2969.192761] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2973.154231] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2974.909794] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2978.247764] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2978.399192] Lustre: lustre-OST0000: new disk, initializing [ 2978.401939] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2980.955351] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2984.158113] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2985.869450] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2987.018075] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2987.025127] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2987.080830] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2988.053498] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 2989.754050] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2989.882966] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2992.100472] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2992.101580] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2992.104101] LustreError: Skipped 1 previous similar message [ 2992.109199] Lustre: Skipped 1 previous similar message [ 2992.110046] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2992.116768] Lustre: Skipped 8 previous similar messages [ 2996.884850] Lustre: server umount lustre-OST0000 complete [ 3004.694775] Lustre: server umount lustre-MDT0000 complete [ 3006.084059] LustreError: 195255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772170279 with bad export cookie 2938202566012024236 [ 3006.090112] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3006.092058] LustreError: 195255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3006.244455] Lustre: server umount lustre-MDT0001 complete [ 3009.737105] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3009.937596] LustreError: 199090:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3009.944482] LustreError: 199090:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 3011.509993] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3014.417460] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3016.129217] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3018.294696] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3019.852075] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3022.878465] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3025.164490] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3027.322136] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3029.105262] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3035.389976] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 6 sec [ 3037.140476] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3037.236096] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3039.716284] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3040.044735] Lustre: server umount lustre-OST0000 complete [ 3041.999575] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3042.074183] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3043.391819] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3043.476265] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3067.721722] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3067.860228] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3067.862625] Lustre: Skipped 5 previous similar messages [ 3070.057203] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3072.131580] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3072.907605] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3076.259250] Lustre: server umount lustre-OST0000 complete [ 3078.113552] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3078.114355] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3078.121449] Lustre: Skipped 6 previous similar messages [ 3078.125289] Lustre: Skipped 4 previous similar messages [ 3080.184188] Lustre: server umount lustre-MDT0000 complete [ 3081.185543] LustreError: 199801:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3081.194847] LustreError: 199801:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3082.009712] LustreError: 199069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772170355 with bad export cookie 2938202566012025104 [ 3082.016734] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3082.017085] LustreError: 199069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3082.174813] Lustre: server umount lustre-MDT0001 complete [ 3085.750309] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 00:32:39 (1772170359) [ 3088.591799] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3090.639403] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3094.073690] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3095.831594] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3098.502029] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3100.234089] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3103.347667] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3106.007952] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3108.480980] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3110.317420] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3114.573967] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 3116.199434] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3116.294475] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3118.890485] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3118.976226] Lustre: lustre-OST0001: new disk, initializing [ 3118.978647] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3121.507412] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3125.031340] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3125.780968] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3125.785062] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3125.832629] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3126.649409] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3126.775284] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3128.496409] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3128.620277] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3130.850509] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3130.855901] LustreError: Skipped 1 previous similar message [ 3135.439142] Lustre: server umount lustre-OST0001 complete [ 3137.955098] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3138.021539] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3139.722346] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3139.815561] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3164.056165] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3166.309630] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3168.554724] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3169.232889] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3169.253452] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3169.256776] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3169.256822] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3169.261111] Lustre: Skipped 1 previous similar message [ 3172.097465] Lustre: server umount lustre-OST0001 complete [ 3191.027415] Lustre: server umount lustre-OST0000 complete [ 3199.219772] Lustre: server umount lustre-MDT0000 complete [ 3200.730485] LustreError: 207636:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772170474 with bad export cookie 2938202566012026035 [ 3200.736122] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3200.736485] LustreError: 207636:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3215.327244] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3215.442356] Lustre: server umount lustre-MDT0001 complete [ 3224.837176] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 00:34:58 (1772170498) [ 3227.699347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3227.877764] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3227.897355] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3228.016151] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3228.019984] Lustre: Skipped 5 previous similar messages [ 3229.722645] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3233.633622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3233.690073] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3233.706640] Lustre: 212580:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3235.551196] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3237.960680] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3239.568221] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3242.594257] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3242.665831] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3244.767324] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3247.102758] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3252.211574] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3252.260795] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3254.871917] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3257.425263] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3258.354135] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3260.378251] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3260.520781] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3262.501218] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3262.586856] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3264.287984] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8cbd19cd7800.ost_server_uuid 50 [ 3265.100886] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8cbd19cd7800.ost_server_uuid in FULL state after 0 sec [ 3265.659905] Lustre: Permanently deactivating lustre-OST0001 [ 3265.669929] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3266.760945] Lustre: Permanently reactivating lustre-OST0001 [ 3268.073327] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3268.076643] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3268.079702] Lustre: Skipped 8 previous similar messages [ 3268.084526] LustreError: Skipped 5 previous similar messages [ 3268.087647] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3268.092373] Lustre: Skipped 10 previous similar messages [ 3274.002147] Lustre: server umount lustre-OST0001 complete [ 3278.309536] LustreError: 214199:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3278.320355] LustreError: 214199:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 31 previous similar messages [ 3282.027663] Lustre: server umount lustre-OST0000 complete [ 3289.832361] Lustre: server umount lustre-MDT0000 complete [ 3291.680156] LustreError: 212571:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772170565 with bad export cookie 2938202566012027309 [ 3291.691184] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3291.839166] Lustre: server umount lustre-MDT0001 complete [ 3302.515625] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 00:36:16 (1772170576) [ 3305.792704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3305.907565] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3305.921284] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3305.925604] Lustre: Skipped 5 previous similar messages [ 3307.715985] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3311.076590] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3311.110661] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3311.120723] Lustre: 218682:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3312.478788] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3314.770117] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3316.310477] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3319.414726] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3319.545147] Lustre: Permanently deactivating lustre-OST0000 [ 3319.554812] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 3322.102338] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3323.901271] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3325.446571] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3326.622308] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3328.705455] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3330.804751] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3331.126498] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 3337.001087] Lustre: Permanently reactivating lustre-OST0000 [ 3337.010944] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 3337.015374] Lustre: Skipped 2 previous similar messages [ 3339.703312] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 3339.706822] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3339.709940] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3339.727113] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 3339.735659] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3348.167578] Lustre: server umount lustre-OST0001 complete [ 3351.894722] Lustre: server umount lustre-OST0000 complete [ 3359.660205] Lustre: server umount lustre-MDT0000 complete [ 3361.368824] LustreError: 218674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772170635 with bad export cookie 2938202566012028975 [ 3361.374966] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3361.375512] LustreError: 218674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 3361.519027] Lustre: server umount lustre-MDT0001 complete [ 3365.414812] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 3366.470973] Key type lgssc unregistered [ 3366.619693] LNet: 223062:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3366.627891] LNetError: 223062:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3366.638452] LNet: Removed LNI 192.168.204.154@tcp [ 3367.102446] Key type .llcrypt unregistered [ 3367.105132] Key type ._llcrypt unregistered [ 3378.696968] Key type ._llcrypt registered [ 3378.697911] Key type .llcrypt registered [ 3378.736403] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 3379.055906] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3379.099087] alg: No test for adler32 (adler32-zlib) [ 3379.949238] Lustre: Lustre: Build Version: 2.17.50_152_g82f9dcf [ 3380.027157] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 3381.607204] Key type lgssc registered [ 3382.167905] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3386.271630] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3386.281651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3387.457239] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3388.836365] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3392.056308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3392.095167] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3392.103159] Lustre: 224479:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3392.106213] Lustre: Permanently deactivating lustre-MDT0001 [ 3392.110380] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3392.205586] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3393.777279] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3394.165353] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3394.169831] LustreError: 224520:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3394.177359] LustreError: 224520:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3394.184220] LustreError: 224520:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 1, failed: rc = -108 [ 3396.541009] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3398.391622] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3401.574746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3401.736243] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3402.787714] LustreError: 226106:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3402.794681] LustreError: 226106:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3404.150624] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3406.488845] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3407.843660] LustreError: 226106:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3408.956157] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3409.075442] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3409.115311] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 3411.627825] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3412.203311] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 3412.204372] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 3413.778714] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3415.613927] Lustre: Permanently reactivating lustre-MDT0001 [ 3415.624500] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 3415.628856] Lustre: Skipped 2 previous similar messages [ 3416.302912] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3421.874859] Lustre: lustre-MDT0001: Client 9746c5a7-7fb2-4556-a4be-2ffd0256c38c (at 192.168.204.54@tcp) reconnecting [ 3423.602331] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3424.525043] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3426.416598] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 3426.477032] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 3428.063679] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 3428.179458] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 3429.396619] Lustre: Permanently deactivating lustre-MDT0001 [ 3429.402962] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3429.406338] Lustre: Skipped 2 previous similar messages [ 3438.096640] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3447.268106] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3447.271475] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3447.277527] Lustre: Skipped 2 previous similar messages [ 3447.279420] Lustre: Skipped 1 previous similar message [ 3450.144782] Lustre: server umount lustre-MDT0000 complete [ 3451.828177] LustreError: 227588:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772170725 with bad export cookie 15386464745179392995 [ 3451.834727] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3451.834984] LustreError: 227588:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3451.955418] Lustre: server umount lustre-MDT0001 complete [ 3459.657319] Lustre: server umount lustre-OST0000 complete [ 3464.631477] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 00:38:58 (1772170738) [ 3469.613622] Lustre: server umount lustre-OST0001 complete [ 3472.758325] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 3477.575570] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 3493.767373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3493.920752] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3493.928547] Lustre: Skipped 2 previous similar messages [ 3493.945463] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3494.002345] Lustre: lustre-MDT0000: new disk, initializing [ 3494.027293] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3494.033402] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3495.542010] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3500.052418] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3500.109978] Lustre: 232176:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3500.147677] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3500.152127] Lustre: Skipped 1 previous similar message [ 3500.203992] Lustre: lustre-MDT0001: new disk, initializing [ 3500.278647] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3500.285914] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3501.942630] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3505.028725] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3506.361665] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3509.674747] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3509.822466] Lustre: lustre-OST0000: new disk, initializing [ 3509.825480] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3509.857538] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3509.859486] Lustre: Skipped 1 previous similar message [ 3512.293922] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3516.300375] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3517.963146] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3517.970892] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3518.015125] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3522.400128] LustreError: 232185:0:(mdt_handler.c:3252:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3527.700706] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3527.781563] Lustre: lustre-OST0001: new disk, initializing [ 3527.783944] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3527.823388] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3530.384600] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3532.487135] LustreError: 232185:0:(mdt_handler.c:3252:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3532.497549] LustreError: 232185:0:(mdt_handler.c:3252:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3534.367371] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3535.170324] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3535.897747] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3535.905334] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3535.956123] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3542.591132] LustreError: 232185:0:(mdt_handler.c:3252:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3544.245389] Lustre: lustre-OST0001: Not available for connect from 192.168.204.54@tcp (stopping) [ 3544.250910] Lustre: Skipped 2 previous similar messages [ 3546.088934] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3546.089319] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3546.093427] LustreError: Skipped 1 previous similar message [ 3546.102207] Lustre: Skipped 1 previous similar message [ 3546.106126] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3546.113346] Lustre: Skipped 1 previous similar message [ 3549.352253] Lustre: lustre-OST0001: Not available for connect from 192.168.204.54@tcp (stopping) [ 3549.645505] Lustre: server umount lustre-OST0001 complete [ 3551.206737] LustreError: 235577:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3551.218322] LustreError: 235577:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3556.325836] LustreError: 233904:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3556.334981] LustreError: 233904:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3561.448206] LustreError: 235784:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3561.461479] LustreError: 235784:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3564.001138] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3564.001183] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3564.006148] LustreError: Skipped 1 previous similar message [ 3564.015739] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3564.021900] Lustre: Skipped 1 previous similar message [ 3568.554182] Lustre: server umount lustre-OST0000 complete [ 3571.682752] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3571.690380] Lustre: Skipped 2 previous similar messages [ 3576.625046] Lustre: server umount lustre-MDT0000 complete [ 3576.800844] LustreError: 232188:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3576.811510] LustreError: 232188:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3578.385086] LustreError: 232169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772170852 with bad export cookie 15386464745179395802 [ 3578.391938] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3578.392662] LustreError: 232169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3584.611431] Lustre: server umount lustre-MDT0001 complete [ 3589.633379] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 3590.783309] Key type lgssc unregistered [ 3590.941531] LNet: 236826:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3590.947830] LNetError: 236826:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3590.959540] LNet: Removed LNI 192.168.204.154@tcp [ 3591.344231] Key type .llcrypt unregistered [ 3591.345359] Key type ._llcrypt unregistered [ 3609.525300] Key type ._llcrypt registered [ 3609.526349] Key type .llcrypt registered [ 3609.567553] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 3609.905751] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3609.941719] alg: No test for adler32 (adler32-zlib) [ 3610.826589] Lustre: Lustre: Build Version: 2.17.50_152_g82f9dcf [ 3610.930876] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 3612.519106] Key type lgssc registered [ 3612.857252] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3616.072225] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3616.083493] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3617.212695] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3617.224954] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3617.295285] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3619.048615] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3623.421215] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3623.474686] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3623.492919] Lustre: 238993:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3623.662512] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3625.724376] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3628.566666] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3630.454701] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3633.735222] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3633.831666] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3633.890305] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3635.682639] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3637.427263] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3641.400861] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 3687.392600] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3687.394396] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3687.394941] LustreError: Skipped 1 previous similar message [ 3687.398423] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3687.401303] Lustre: Skipped 1 previous similar message [ 3688.703099] Lustre: server umount lustre-OST0000 complete [ 3690.555314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 3696.291373] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3696.478523] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3698.932841] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3701.523938] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3707.754405] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3707.761828] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3707.761995] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3707.773083] Lustre: Skipped 1 previous similar message [ 3738.597539] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3738.598180] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3738.602064] LustreError: Skipped 1 previous similar message [ 3738.610758] Lustre: Skipped 1 previous similar message [ 3738.614360] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3738.620308] Lustre: Skipped 1 previous similar message [ 3741.755040] Lustre: server umount lustre-OST0000 complete [ 3743.714396] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3743.715338] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3743.723040] Lustre: Skipped 1 previous similar message [ 3743.727455] Lustre: Skipped 1 previous similar message [ 3748.832598] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3748.835393] Lustre: Skipped 1 previous similar message [ 3749.817202] Lustre: server umount lustre-MDT0000 complete [ 3751.660909] LustreError: 238984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772171025 with bad export cookie 9321248915533146341 [ 3751.666912] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3751.824838] Lustre: server umount lustre-MDT0001 complete [ 3756.867285] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 3758.546372] Key type lgssc unregistered [ 3758.740464] LNet: 243945:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3758.745522] LNetError: 243945:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3758.754814] LNet: Removed LNI 192.168.204.154@tcp [ 3759.260338] Key type .llcrypt unregistered [ 3759.262096] Key type ._llcrypt unregistered [ 3771.404841] Key type ._llcrypt registered [ 3771.406599] Key type .llcrypt registered [ 3771.464645] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 3771.870844] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3771.884354] alg: No test for adler32 (adler32-zlib) [ 3772.779058] Lustre: Lustre: Build Version: 2.17.50_152_g82f9dcf [ 3772.899350] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 3774.495333] Key type lgssc registered [ 3775.105794] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3778.676475] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3778.686348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3779.877817] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3781.222638] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3784.931388] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3785.096676] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3786.719293] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3788.834485] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3790.693550] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3794.171783] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3794.396242] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3796.757411] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3799.342745] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3804.339808] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 3817.063878] Lustre: server umount lustre-OST0000 complete [ 3820.002868] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3820.005258] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3820.011224] Lustre: Skipped 1 previous similar message [ 3825.121463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3825.125243] Lustre: Skipped 2 previous similar messages [ 3828.234950] Lustre: server umount lustre-MDT0000 complete [ 3830.093940] LustreError: 245302:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772171103 with bad export cookie 10634473637918394693 [ 3830.100921] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3830.103328] LustreError: 245302:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3839.540693] Lustre: server umount lustre-MDT0001 complete [ 3844.173262] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 3845.034605] Key type lgssc unregistered [ 3845.176674] LNet: 249274:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3845.182057] LNetError: 249274:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3845.192500] LNet: Removed LNI 192.168.204.154@tcp [ 3845.635530] Key type .llcrypt unregistered [ 3845.637920] Key type ._llcrypt unregistered [ 3853.964755] Key type ._llcrypt registered [ 3853.966400] Key type .llcrypt registered [ 3854.026212] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 3854.377675] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3854.423837] alg: No test for adler32 (adler32-zlib) [ 3855.284974] Lustre: Lustre: Build Version: 2.17.50_152_g82f9dcf [ 3855.371772] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 3856.967214] Key type lgssc registered [ 3857.503336] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3861.073231] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3861.084874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3862.292992] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3864.059115] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3868.459178] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3868.679260] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3870.699126] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3872.820586] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3874.365051] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3877.807796] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3878.023741] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3880.597348] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3882.635798] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3887.278620] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 3894.752568] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3894.757460] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3894.765871] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3897.826564] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3897.829238] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3897.831640] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3902.947602] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3902.951300] Lustre: Skipped 2 previous similar messages [ 3907.039184] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3907.126551] Lustre: server umount lustre-OST0000 complete [ 3909.600387] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3909.604689] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3909.610731] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3913.186536] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3914.848976] Lustre: server umount lustre-MDT0000 complete [ 3916.628972] LustreError: 250439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772171190 with bad export cookie 10744094367567060887 [ 3916.634447] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3916.637888] LustreError: 250439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3916.804266] Lustre: server umount lustre-MDT0001 complete [ 3921.798299] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 3923.036033] Key type lgssc unregistered [ 3923.203629] LNet: 253782:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3923.208473] LNetError: 253782:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3923.217682] LNet: Removed LNI 192.168.204.154@tcp [ 3923.710053] Key type .llcrypt unregistered [ 3923.712200] Key type ._llcrypt unregistered [ 3935.201509] Key type ._llcrypt registered [ 3935.203495] Key type .llcrypt registered [ 3935.272436] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 3935.689646] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3935.746566] alg: No test for adler32 (adler32-zlib) [ 3936.606494] Lustre: Lustre: Build Version: 2.17.50_152_g82f9dcf [ 3936.691141] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 3938.287271] Key type lgssc registered [ 3938.786850] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3942.323548] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3942.335366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3943.557175] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3945.304910] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3949.532894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3949.659277] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3951.345220] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3954.152539] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3956.069691] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3959.534130] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3959.714826] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3962.463738] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3965.083717] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3968.942408] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3981.280886] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3981.286452] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3981.296225] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3984.360528] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3984.362169] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3984.362351] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3987.246279] Lustre: server umount lustre-OST0000 complete [ 3989.474014] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3989.475476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3989.482460] Lustre: Skipped 1 previous similar message [ 3989.489573] Lustre: Skipped 2 previous similar messages [ 3994.593972] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3994.597991] Lustre: Skipped 1 previous similar message [ 3995.324320] Lustre: server umount lustre-MDT0000 complete [ 3996.996750] LustreError: 255138:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772171270 with bad export cookie 2285031828302365907 [ 3997.001723] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3997.003493] LustreError: 255138:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3997.010056] LustreError: 254473:0:(import.c:716:ptlrpc_connect_import_locked()) already connecting [ 3997.139623] Lustre: server umount lustre-MDT0001 complete [ 4002.120335] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 4003.614714] Key type lgssc unregistered [ 4003.763658] LNet: 259117:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4003.768739] LNetError: 259117:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4003.781554] LNet: Removed LNI 192.168.204.154@tcp [ 4004.212674] Key type .llcrypt unregistered [ 4004.213646] Key type ._llcrypt unregistered [ 4013.254563] Key type ._llcrypt registered [ 4013.256344] Key type .llcrypt registered [ 4013.314073] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 4013.670570] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4013.703448] alg: No test for adler32 (adler32-zlib) [ 4014.544366] Lustre: Lustre: Build Version: 2.17.50_152_g82f9dcf [ 4014.621543] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 4016.207199] Key type lgssc registered [ 4016.706324] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4020.059586] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4020.070949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4021.265570] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4022.987276] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4026.934528] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4027.080983] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4028.765883] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4030.811148] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4032.455089] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4035.345364] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4035.536932] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4038.346558] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4040.597227] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4046.831155] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 4057.065150] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4057.068282] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4057.072075] LustreError: Skipped 1 previous similar message [ 4057.073259] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4057.080244] Lustre: Skipped 1 previous similar message [ 4057.089137] Lustre: Skipped 1 previous similar message [ 4061.262107] Lustre: server umount lustre-OST0000 complete [ 4063.200725] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4063.205583] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4063.214070] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4067.297708] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4067.299317] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4069.175108] Lustre: server umount lustre-MDT0000 complete [ 4070.784021] LustreError: 260279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772171344 with bad export cookie 11656050902258534817 [ 4070.788151] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4070.790330] LustreError: 260279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4070.932533] Lustre: server umount lustre-MDT0001 complete [ 4075.898181] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 4077.469471] Key type lgssc unregistered [ 4077.643662] LNet: 263675:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4077.648727] LNetError: 263675:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4077.658466] LNet: Removed LNI 192.168.204.154@tcp [ 4078.086515] Key type .llcrypt unregistered [ 4078.088758] Key type ._llcrypt unregistered [ 4082.896412] Key type ._llcrypt registered [ 4082.898619] Key type .llcrypt registered [ 4082.949616] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 4090.311733] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 4097.919646] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 4098.231043] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4098.242179] alg: No test for adler32 (adler32-zlib) [ 4099.112313] Lustre: Lustre: Build Version: 2.17.50_152_g82f9dcf [ 4099.214135] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 4100.799170] Key type lgssc registered [ 4101.423708] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4118.088868] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4118.110440] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4119.244299] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4119.264941] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4119.314421] Lustre: lustre-MDT0000: new disk, initializing [ 4119.342919] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4119.349495] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4121.142624] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4126.765595] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4126.841954] Lustre: 267151:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4126.884932] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4126.888564] Lustre: Skipped 1 previous similar message [ 4126.938402] Lustre: lustre-MDT0001: new disk, initializing [ 4126.990774] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4127.012777] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4127.019679] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4128.689918] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4132.062972] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4133.450666] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4135.875576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4135.986207] Lustre: lustre-OST0000: new disk, initializing [ 4135.988218] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4136.017759] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4138.276062] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4139.261242] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4139.265622] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4139.303681] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4141.413153] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4143.065069] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4143.160385] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4144.797768] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4144.884843] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4149.729654] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4149.730306] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4149.731420] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4149.733681] LustreError: Skipped 1 previous similar message [ 4149.737064] Lustre: Skipped 1 previous similar message [ 4154.851396] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4154.856600] Lustre: Skipped 2 previous similar messages [ 4159.967154] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4159.971569] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4159.980782] Lustre: Skipped 1 previous similar message [ 4160.059566] Lustre: server umount lustre-OST0000 complete [ 4163.040574] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4163.045629] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4163.054431] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4165.090751] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4167.542992] Lustre: server umount lustre-MDT0000 complete [ 4168.812420] LustreError: 267142:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772171442 with bad export cookie 12183725268371411430 [ 4168.815798] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4168.816188] LustreError: 267142:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4168.931875] Lustre: server umount lustre-MDT0001 complete [ 4172.337951] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 00:50:46 (1772171446) [ 4175.492588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4175.747580] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4177.152215] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4180.635253] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4180.772607] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4182.406754] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4184.604302] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4186.221792] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4189.178767] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4189.393964] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4192.259661] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4194.587540] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4211.173293] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4211.174883] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4211.176587] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4211.176595] Lustre: Skipped 2 previous similar messages [ 4211.192590] Lustre: Skipped 1 previous similar message [ 4214.763086] Lustre: server umount lustre-OST0000 complete [ 4216.288262] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4216.291935] Lustre: Skipped 1 previous similar message [ 4221.409165] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4221.413218] Lustre: Skipped 4 previous similar messages [ 4222.453558] Lustre: server umount lustre-MDT0000 complete [ 4224.265799] LustreError: 271140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772171498 with bad export cookie 12183725268371412319 [ 4224.273098] LustreError: 271140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4224.274088] LustreError: 271877:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000bd2:0x1:0x0]: rc = -5 [ 4224.275327] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4224.464705] Lustre: server umount lustre-MDT0001 complete [ 4229.307328] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 4230.900974] Key type lgssc unregistered [ 4231.078852] LNet: 274359:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4231.084224] LNetError: 274359:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4231.093802] LNet: Removed LNI 192.168.204.154@tcp [ 4231.568996] Key type .llcrypt unregistered [ 4231.570762] Key type ._llcrypt unregistered [ 4242.210402] Key type ._llcrypt registered [ 4242.213140] Key type .llcrypt registered [ 4242.269582] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 4249.644550] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 4250.093450] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4250.102543] alg: No test for adler32 (adler32-zlib) [ 4250.975949] Lustre: Lustre: Build Version: 2.17.50_152_g82f9dcf [ 4251.080707] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 4252.663175] Key type lgssc registered [ 4253.259741] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4272.627827] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4272.637886] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4273.754717] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4273.766440] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4273.804606] Lustre: lustre-MDT0000: new disk, initializing [ 4273.830214] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4273.838208] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4275.248191] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4280.240217] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4280.278960] Lustre: 278122:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4280.301779] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4280.304721] Lustre: Skipped 1 previous similar message [ 4280.345144] Lustre: lustre-MDT0001: new disk, initializing [ 4280.379681] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4280.393849] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4280.396952] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4281.927737] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4285.662031] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4287.316498] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4290.142077] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4290.271902] Lustre: lustre-OST2710: new disk, initializing [ 4290.273986] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 4290.311363] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 4292.832483] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4296.929459] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 4298.253659] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 4298.260608] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 4298.314618] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 4300.162235] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4300.267609] Lustre: lustre-OST03e8: new disk, initializing [ 4300.271177] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 4300.318306] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 4302.394224] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4304.959067] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 4305.911130] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 4305.913788] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 4305.939831] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 4308.109764] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 4308.219181] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4310.037525] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 4310.161818] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4311.984268] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 4312.090389] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4313.786536] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 4313.883202] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4316.131968] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4316.136439] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4316.136865] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4316.141862] Lustre: Skipped 2 previous similar messages [ 4321.250357] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4321.254863] Lustre: Skipped 5 previous similar messages [ 4321.529047] Lustre: server umount lustre-MDT0000 complete [ 4323.393862] LustreError: 280813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772171597 with bad export cookie 16463282846263502186 [ 4323.401420] LustreError: 280813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4323.401526] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4323.540583] Lustre: server umount lustre-MDT0001 complete [ 4335.633787] Lustre: server umount lustre-OST2710 complete [ 4347.706484] Lustre: server umount lustre-OST03e8 complete [ 4356.223302] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 4360.942427] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 4378.691053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4378.831904] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4378.851243] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4378.913662] Lustre: lustre-MDT0000: new disk, initializing [ 4378.955959] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4378.968552] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4380.803945] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4386.051368] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4386.105996] Lustre: 285844:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4386.306949] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4388.230965] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4392.101538] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4394.017489] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4397.448666] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4397.615742] Lustre: lustre-OST0000: new disk, initializing [ 4397.618906] Lustre: Skipped 1 previous similar message [ 4397.623635] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4397.627821] Lustre: Skipped 2 previous similar messages [ 4397.679980] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4397.684493] Lustre: Skipped 1 previous similar message [ 4400.120249] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4403.212208] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4403.217081] Lustre: Skipped 1 previous similar message [ 4403.220267] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4403.267790] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4404.253902] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4406.209601] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4406.306875] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4408.117743] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4408.212042] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4413.413957] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4413.414238] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4413.421897] Lustre: Skipped 1 previous similar message [ 4413.427134] LustreError: Skipped 1 previous similar message [ 4413.428281] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4413.437921] Lustre: Skipped 1 previous similar message [ 4415.276573] Lustre: server umount lustre-OST0000 complete [ 4418.532043] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4418.533465] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4418.539959] Lustre: Skipped 1 previous similar message [ 4418.546809] Lustre: Skipped 1 previous similar message [ 4423.365307] Lustre: server umount lustre-MDT0000 complete [ 4425.344198] LustreError: 285835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772171699 with bad export cookie 16463282846263503838 [ 4425.351602] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4425.352701] LustreError: 285835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4425.504893] Lustre: server umount lustre-MDT0001 complete [ 4429.923182] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 00:55:03 (1772171703) [ 4430.777262] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 4431.742677] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 00:55:05 (1772171705) [ 4433.283541] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 4446.565869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4446.684199] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4446.692169] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4446.758460] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4448.181279] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4452.033356] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4452.086145] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4452.099122] Lustre: 291277:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4453.555923] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4455.754861] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4457.227712] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4471.623580] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4471.751130] Lustre: Denying initial registration attempt from nid 192.168.204.154@tcp, specified as failover [ 4471.757850] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 4471.763634] LustreError: 292889:0:(tgt_mount.c:2579:server_fill_super()) Unable to start targets: -99 [ 4471.768374] LustreError: 292889:0:(tgt_mount.c:2066:server_put_super()) no obd lustre-OST0000 [ 4471.772375] LustreError: 292889:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 4471.814390] Lustre: server umount lustre-OST0000 complete [ 4471.817436] LustreError: 292889:0:(super25.c:190:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 4474.849806] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4474.850744] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4474.859815] Lustre: Skipped 1 previous similar message [ 4480.688645] Lustre: server umount lustre-MDT0000 complete [ 4482.327622] LustreError: 292000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772171756 with bad export cookie 16463282846263504706 [ 4482.333732] LustreError: 292000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4482.334726] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4482.491111] Lustre: server umount lustre-MDT0001 complete [ 4487.594540] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 4489.212419] Key type lgssc unregistered [ 4489.376660] LNet: 293991:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4489.379447] LNetError: 293991:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4489.385598] LNet: Removed LNI 192.168.204.154@tcp [ 4489.808934] Key type .llcrypt unregistered [ 4489.810884] Key type ._llcrypt unregistered [ 4494.451731] Key type ._llcrypt registered [ 4494.452806] Key type .llcrypt registered [ 4494.495861] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 4494.893915] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4494.959041] alg: No test for adler32 (adler32-zlib) [ 4495.868970] Lustre: Lustre: Build Version: 2.17.50_152_g82f9dcf [ 4496.000347] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 4497.615269] Key type lgssc registered [ 4498.220938] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4538.287746] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 4541.814436] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4541.826045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4542.959904] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4542.972607] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4543.048111] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4544.913370] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4548.702085] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4548.752570] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4548.769819] Lustre: 296704:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4548.930581] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4550.621289] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4552.919960] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4554.206444] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4568.056126] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4568.181955] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4568.303910] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4570.945237] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4573.237206] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4574.308490] Lustre: server umount lustre-OST0000 complete [ 4577.789850] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4577.791476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4577.796471] Lustre: Skipped 1 previous similar message [ 4581.865745] Lustre: server umount lustre-MDT0000 complete [ 4583.301725] LustreError: 296697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772171857 with bad export cookie 15236698689946277861 [ 4583.307792] LustreError: 296697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4583.308419] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4583.445207] Lustre: server umount lustre-MDT0001 complete [ 4587.400662] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 4588.767733] Key type lgssc unregistered [ 4588.914566] LNet: 299910:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4588.918720] LNetError: 299910:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4588.926543] LNet: Removed LNI 192.168.204.154@tcp [ 4589.338205] Key type .llcrypt unregistered [ 4589.339705] Key type ._llcrypt unregistered [ 4599.938097] Key type ._llcrypt registered [ 4599.939091] Key type .llcrypt registered [ 4599.975314] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 4600.400472] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4600.420219] alg: No test for adler32 (adler32-zlib) [ 4601.269689] Lustre: Lustre: Build Version: 2.17.50_152_g82f9dcf [ 4601.346732] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 4602.927284] Key type lgssc registered [ 4603.524960] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4606.590931] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4606.595801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4607.769172] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4609.250250] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4613.202108] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4613.323969] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4614.625235] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4617.127157] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4618.950592] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4621.689907] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4621.819322] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4624.245323] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4626.711910] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4642.483460] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.54@tcp (stopping) [ 4643.298464] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4643.302325] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4643.306461] Lustre: Skipped 2 previous similar messages [ 4643.310470] Lustre: Skipped 1 previous similar message [ 4646.514140] Lustre: server umount lustre-MDT0000 complete [ 4647.589844] LustreError: 301282:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4647.601086] LustreError: 301282:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4648.073349] LustreError: 301268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772171921 with bad export cookie 2151162120110107966 [ 4648.076481] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4648.079960] LustreError: 301268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4648.416461] LustreError: 301282:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4648.416687] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4648.431443] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4652.708692] LustreError: 301282:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4652.715551] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.54@tcp (stopping) [ 4654.289594] Lustre: server umount lustre-MDT0001 complete [ 4656.708385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4659.923394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4672.991532] LustreError: 300597:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000008ea3720f x1858256896024960/t0(0) o250->MGC192.168.204.154@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4673.119699] LustreError: 302895:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4673.124634] LustreError: 302895:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4673.145945] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4673.174864] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [ 4674.579764] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4677.217473] LustreError: 304562:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4677.223365] LustreError: 304562:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4677.882465] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4678.026574] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4679.634557] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4681.813996] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4683.172256] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4684.657463] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 4686.093295] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4686.741675] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4688.358347] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4688.360238] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4688.368127] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4688.377185] Lustre: Skipped 2 previous similar messages [ 4694.289107] Lustre: server umount lustre-OST0000 complete [ 4698.594700] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4698.595006] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4698.610987] Lustre: Skipped 1 previous similar message [ 4698.615707] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4698.621468] Lustre: Skipped 3 previous similar messages [ 4710.367167] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4710.445886] Lustre: server umount lustre-MDT0000 complete [ 4711.810679] LustreError: 302004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772171985 with bad export cookie 2151162120110115134 [ 4711.815954] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4711.817908] LustreError: 302004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4711.953916] Lustre: server umount lustre-MDT0001 complete [ 4715.739251] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 4716.860896] Key type lgssc unregistered [ 4716.991561] LNet: 306992:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4716.995918] LNetError: 306992:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4717.009490] LNet: Removed LNI 192.168.204.154@tcp [ 4717.451896] Key type .llcrypt unregistered [ 4717.453428] Key type ._llcrypt unregistered [ 4728.900221] Key type ._llcrypt registered [ 4728.901152] Key type .llcrypt registered [ 4728.939398] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 4729.316334] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4729.342141] alg: No test for adler32 (adler32-zlib) [ 4730.198521] Lustre: Lustre: Build Version: 2.17.50_152_g82f9dcf [ 4730.281867] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 4731.871241] Key type lgssc registered [ 4732.500406] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4735.669245] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4735.680830] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4736.942426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4738.645355] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4742.608453] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4742.834804] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4744.856365] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4747.506892] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4749.172842] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4752.356621] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4752.358548] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4752.362838] Lustre: Skipped 1 previous similar message [ 4752.369221] Lustre: Skipped 1 previous similar message [ 4753.378329] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4753.382474] Lustre: Skipped 1 previous similar message [ 4758.497135] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4758.499461] Lustre: Skipped 1 previous similar message [ 4763.617480] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4763.622476] Lustre: Skipped 1 previous similar message [ 4765.151115] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4765.210178] Lustre: server umount lustre-MDT0000 complete [ 4766.463146] LustreError: 309088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772172040 with bad export cookie 10641451645983369665 [ 4766.465977] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4766.466136] LustreError: 309088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4766.573186] Lustre: server umount lustre-MDT0001 complete [ 4771.436578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4771.557272] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4771.572368] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4771.667232] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4773.078873] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4776.294289] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4776.325892] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4776.338127] Lustre: 310732:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4776.435079] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4778.000177] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4780.080899] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4781.373420] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4784.487042] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4784.592566] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 4784.596265] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 4784.601313] LustreError: 312388:0:(tgt_mount.c:2579:server_fill_super()) Unable to start targets: -2 [ 4784.605236] LustreError: 312388:0:(tgt_mount.c:2066:server_put_super()) no obd lustre-OST0000 [ 4784.608507] LustreError: 312388:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 4784.641349] Lustre: server umount lustre-OST0000 complete [ 4784.643230] LustreError: 312388:0:(super25.c:190:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 4787.745380] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4787.871780] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4787.971319] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4790.440212] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4793.019582] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4796.380996] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4796.469949] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4796.490963] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [ 4796.550686] Lustre: lustre-OST0001: new disk, initializing [ 4796.553919] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4796.585744] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4799.143111] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4801.457634] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4802.669774] Lustre: server umount lustre-OST0001 complete [ 4806.140771] LustreError: 312852:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4806.147600] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4806.150947] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4806.151033] LustreError: 312852:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4806.177113] Lustre: Skipped 1 previous similar message [ 4810.386829] Lustre: server umount lustre-OST0000 complete [ 4812.256275] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4812.259420] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4816.353365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4816.354475] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4816.357342] Lustre: Skipped 2 previous similar messages [ 4818.139542] Lustre: server umount lustre-MDT0000 complete [ 4819.856317] LustreError: 310723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772172093 with bad export cookie 10641451645983370204 [ 4819.863355] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4819.863735] LustreError: 310723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4820.021543] Lustre: server umount lustre-MDT0001 complete [ 4824.591558] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 4826.209655] Key type lgssc unregistered [ 4826.378607] LNet: 315647:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4826.383563] LNetError: 315647:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4826.393716] LNet: Removed LNI 192.168.204.154@tcp [ 4826.889209] Key type .llcrypt unregistered [ 4826.891354] Key type ._llcrypt unregistered [ 4865.837232] Key type ._llcrypt registered [ 4865.838371] Key type .llcrypt registered [ 4865.884973] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 4873.434662] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 4873.990427] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4874.009941] alg: No test for adler32 (adler32-zlib) [ 4874.857715] Lustre: Lustre: Build Version: 2.17.50_152_g82f9dcf [ 4874.933096] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 4876.511233] Key type lgssc registered [ 4877.025139] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4894.282529] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4894.295069] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4895.440194] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4895.459951] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4895.515657] Lustre: lustre-MDT0000: new disk, initializing [ 4895.552641] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4895.564622] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4897.328328] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4902.819131] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4902.887595] Lustre: 320516:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4902.930060] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4902.934687] Lustre: Skipped 1 previous similar message [ 4902.993017] Lustre: lustre-MDT0001: new disk, initializing [ 4903.037776] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4903.056107] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4903.062107] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4904.711981] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4908.740481] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4910.475689] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4913.121945] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4913.231436] Lustre: lustre-OST0000: new disk, initializing [ 4913.233326] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4913.260779] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4915.076376] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4918.627257] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4920.321423] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4920.324300] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4920.397322] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4920.507160] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4920.618467] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4922.457966] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4922.571620] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4925.409710] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4925.413902] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4925.414980] LustreError: Skipped 1 previous similar message [ 4925.416136] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4925.422191] Lustre: Skipped 1 previous similar message [ 4929.591371] Lustre: server umount lustre-OST0000 complete [ 4934.112180] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4934.115898] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4934.123508] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4934.126882] Lustre: Skipped 1 previous similar message [ 4935.648591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4935.650590] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4937.651271] Lustre: server umount lustre-MDT0000 complete [ 4940.802080] LustreError: 320508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772172214 with bad export cookie 15255242862418569637 [ 4940.807257] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4940.811394] LustreError: 320508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4941.085889] Lustre: server umount lustre-MDT0001 complete [ 4948.104378] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 01:03:41 (1772172221) [ 4952.386941] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 01:03:46 (1772172226) [ 4955.353508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4955.528216] LustreError: 324863:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4955.536244] LustreError: 324863:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4955.550278] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4956.956904] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4959.896681] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4960.032623] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4961.454439] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4963.391687] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4964.662104] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4966.975870] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4968.954484] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4970.751075] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4984.860887] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 4985.454747] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 4986.031924] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 4986.649342] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 4987.327863] Lustre: Failing over lustre-MDT0000 [ 4987.475818] Lustre: server umount lustre-MDT0000 complete [ 4990.943887] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4990.946114] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4990.950908] LustreError: 324881:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4991.969343] LustreError: 324862:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4991.974523] LustreError: 324862:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4996.772584] LustreError: 324859:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5000.418442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5000.463946] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5000.554619] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5000.557126] Lustre: Skipped 1 previous similar message [ 5000.574175] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5001.783835] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 5001.892853] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5005.796680] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5005.807643] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 5005.823784] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5006.902574] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5007.425413] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5009.249986] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 5009.927784] Lustre: Failing over lustre-MDT0000 [ 5010.113275] Lustre: server umount lustre-MDT0000 complete [ 5010.912486] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5010.913366] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5010.913967] LustreError: 324858:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5010.913974] LustreError: 324858:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5010.925072] Lustre: Skipped 3 previous similar messages [ 5021.152440] LustreError: 324862:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5021.158861] LustreError: 324862:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 5022.971634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5023.008768] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5023.012113] LustreError: 328812:0:(ldlm_resource.c:1170:ldlm_resource_complain()) MGC192.168.204.154@tcp: namespace resource [0x65727473756c:0x0:0x0].0x0 (00000000b2602d39) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 5023.016841] LustreError: 324854:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 5023.114559] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5023.139895] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5024.469840] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 5026.481782] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5028.324677] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5028.327252] Lustre: Skipped 2 previous similar messages [ 5028.334726] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 5028.354578] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 5029.413723] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5029.917257] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5033.442534] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5033.443110] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5033.446582] Lustre: Skipped 1 previous similar message [ 5033.449408] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5033.452161] Lustre: Skipped 3 previous similar messages [ 5038.292776] Lustre: server umount lustre-MDT0000 complete [ 5038.560965] LustreError: 324863:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5038.565599] LustreError: 324863:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5039.623297] LustreError: 324842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772172313 with bad export cookie 15255242862418572157 [ 5039.624992] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5039.629453] LustreError: 324842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5039.747935] Lustre: server umount lustre-MDT0001 complete [ 5050.335175] Lustre: server umount lustre-OST0000 complete [ 5056.119689] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5056.323753] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5057.759593] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 5060.968982] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5062.624889] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 5064.621410] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5066.016263] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5068.535917] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5070.803763] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 5072.832473] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5082.022609] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 5086.175942] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5086.179370] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5086.183403] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5090.769965] Lustre: server umount lustre-OST0000 complete [ 5097.440240] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5097.442575] Lustre: Skipped 5 previous similar messages [ 5098.275025] Lustre: server umount lustre-MDT0000 complete [ 5099.702332] LustreError: 330628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772172373 with bad export cookie 15255242862418572899 [ 5099.709475] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5099.843133] Lustre: server umount lustre-MDT0001 complete [ 5103.777505] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 5104.920980] Key type lgssc unregistered [ 5105.062609] LNet: 333848:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5105.065870] LNetError: 333848:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5105.074450] LNet: Removed LNI 192.168.204.154@tcp [ 5105.405582] Key type .llcrypt unregistered [ 5105.406802] Key type ._llcrypt unregistered [ 5116.028037] Key type ._llcrypt registered [ 5116.029025] Key type .llcrypt registered [ 5116.070739] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 5116.431592] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5116.476586] alg: No test for adler32 (adler32-zlib) [ 5117.346514] Lustre: Lustre: Build Version: 2.17.50_152_g82f9dcf [ 5117.448292] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 5119.039205] Key type lgssc registered [ 5119.454936] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5122.332982] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5122.341557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5123.535631] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5125.028767] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 5128.382419] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5128.552338] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5130.049512] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 5132.111896] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5133.517143] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5136.169282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5136.333909] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5138.539505] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 5140.606956] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5148.646251] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 5158.884341] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5158.885081] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5158.891337] Lustre: Skipped 1 previous similar message [ 5161.631944] Lustre: server umount lustre-MDT0000 complete [ 5163.124234] LustreError: 335307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772172436 with bad export cookie 12311523915019948598 [ 5163.127599] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5163.130221] LustreError: 335307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5163.259649] Lustre: server umount lustre-MDT0001 complete [ 5175.020235] Lustre: server umount lustre-OST0000 complete [ 5180.080204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5180.262763] LustreError: 338720:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5180.271276] LustreError: 338720:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5180.289936] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5181.739925] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 5185.034886] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5185.188128] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5186.650058] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 5188.739772] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5190.251306] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5192.957457] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5195.387630] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 5197.564047] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5205.414421] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5215.713493] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5215.714670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5215.718282] Lustre: Skipped 3 previous similar messages [ 5215.720835] Lustre: Skipped 3 previous similar messages [ 5220.833400] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5220.837017] Lustre: Skipped 2 previous similar messages [ 5221.350867] Lustre: server umount lustre-MDT0000 complete [ 5222.710113] LustreError: 338700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772172496 with bad export cookie 12311523915019949697 [ 5222.713847] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5222.714207] LustreError: 338700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5222.823151] Lustre: server umount lustre-MDT0001 complete [ 5234.527233] Lustre: server umount lustre-OST0000 complete [ 5240.717549] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5240.936392] LustreError: 342312:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5240.966860] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5240.970301] Lustre: Skipped 1 previous similar message [ 5242.553422] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 5246.189940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5247.875616] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 5250.151913] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5251.454635] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5253.912478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5256.147796] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 5258.106913] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5265.317502] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5270.498379] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5270.499417] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5270.500244] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5270.507133] Lustre: Skipped 1 previous similar message [ 5271.948801] Lustre: server umount lustre-OST0000 complete [ 5275.349151] Lustre: server umount lustre-MDT0000 complete [ 5275.616418] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5275.617180] LustreError: 342306:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5275.621467] Lustre: Skipped 1 previous similar message [ 5276.623195] LustreError: 342293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772172550 with bad export cookie 12311523915019950705 [ 5276.626032] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5276.627409] LustreError: 342293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5276.737756] Lustre: server umount lustre-MDT0001 complete [ 5280.161568] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 5281.167685] Key type lgssc unregistered [ 5281.288400] LNet: 345508:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5281.291257] LNetError: 345508:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5281.300301] LNet: Removed LNI 192.168.204.154@tcp [ 5281.598739] Key type .llcrypt unregistered [ 5281.602063] Key type ._llcrypt unregistered [ 5290.725326] Key type ._llcrypt registered [ 5290.726260] Key type .llcrypt registered [ 5290.762135] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 5291.196557] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5291.216294] alg: No test for adler32 (adler32-zlib) [ 5292.055922] Lustre: Lustre: Build Version: 2.17.50_152_g82f9dcf [ 5292.129539] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 5293.703143] Key type lgssc registered [ 5294.040933] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5296.321132] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5296.326161] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5296.329446] LustreError: 346907:0:(osd_handler.c:8556:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 5296.339058] LustreError: 346907:0:(obd_config.c:880:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 5296.341683] LustreError: 346907:0:(obd_mount.c:217:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 5296.344091] LustreError: 346907:0:(tgt_mount.c:2547:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 5296.346624] LustreError: 346907:0:(super25.c:190:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5298.674123] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5298.677203] LustreError: 347303:0:(osd_handler.c:8556:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 5298.686185] LustreError: 347303:0:(obd_config.c:880:class_setup()) setup lustre-OST0000-osd failed (-22) [ 5298.688460] LustreError: 347303:0:(obd_mount.c:217:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 5298.690417] LustreError: 347303:0:(tgt_mount.c:2547:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 5298.692608] LustreError: 347303:0:(super25.c:190:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5304.581162] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 5305.573961] Key type lgssc unregistered [ 5305.697265] LNet: 348288:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5305.699291] LNetError: 348288:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5305.710327] LNet: Removed LNI 192.168.204.154@tcp [ 5305.979412] Key type .llcrypt unregistered [ 5305.980339] Key type ._llcrypt unregistered [ 5311.445093] Key type ._llcrypt registered [ 5311.445919] Key type .llcrypt registered [ 5311.481951] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 5317.347566] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 5317.632406] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5317.639032] alg: No test for adler32 (adler32-zlib) [ 5318.488246] Lustre: Lustre: Build Version: 2.17.50_152_g82f9dcf [ 5318.568624] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 5320.151188] Key type lgssc registered [ 5320.503620] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5331.442478] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5331.448066] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5332.538250] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5332.551334] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5332.589184] Lustre: lustre-MDT0000: new disk, initializing [ 5332.613163] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5332.619891] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5333.849557] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 5337.968232] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5338.015118] Lustre: 351384:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5338.046389] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 5338.049328] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5338.051072] Lustre: Skipped 1 previous similar message [ 5338.085334] Lustre: lustre-MDT0001: new disk, initializing [ 5338.117651] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5338.130351] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5338.133909] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5339.395646] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 5342.420579] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5343.561457] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5345.711084] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5345.805260] Lustre: lustre-OST0000: new disk, initializing [ 5345.807192] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5345.833185] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5347.062026] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5347.065800] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5347.096566] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5347.589415] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 5350.567065] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5351.772140] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5351.836600] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5353.096210] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5353.163803] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5357.536632] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5357.537587] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5357.539564] LustreError: Skipped 1 previous similar message [ 5357.540121] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5357.543952] Lustre: Skipped 1 previous similar message [ 5362.657930] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5362.659738] Lustre: Skipped 2 previous similar messages [ 5367.778753] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5367.780923] Lustre: Skipped 1 previous similar message [ 5368.287150] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5368.349156] Lustre: server umount lustre-OST0000 complete [ 5372.897327] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5372.898949] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5372.902023] Lustre: Skipped 1 previous similar message [ 5375.778174] Lustre: server umount lustre-MDT0000 complete [ 5377.035150] LustreError: 351374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772172650 with bad export cookie 7586349078889918760 [ 5377.037991] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5377.038337] LustreError: 351374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5377.153112] Lustre: server umount lustre-MDT0001 complete [ 5379.918807] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 5380.469764] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 01:10:54 (1772172654) [ 5382.700811] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5382.869282] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5384.200303] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 5387.118693] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5387.256506] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5388.653301] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 5390.491753] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5391.676058] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5393.898579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5395.828070] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 5397.818322] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5400.584390] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5400.695634] Lustre: lustre-OST0001: new disk, initializing [ 5400.699673] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5400.784770] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5400.788559] Lustre: Skipped 1 previous similar message [ 5401.828183] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5401.835592] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5401.911301] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5402.707091] hrtimer: interrupt took 6078182 ns [ 5404.818181] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 5411.734282] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5417.446512] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5417.448408] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5417.451397] LustreError: Skipped 1 previous similar message [ 5417.452867] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5417.452883] Lustre: Skipped 1 previous similar message [ 5417.486767] Lustre: Skipped 1 previous similar message [ 5422.890500] Lustre: server umount lustre-OST0001 complete [ 5423.588934] LustreError: 358056:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5423.616224] LustreError: 358056:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5424.809369] LustreError: 358056:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5428.720899] LustreError: 357028:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5428.732945] LustreError: 357028:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5433.829044] LustreError: 358056:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5433.852578] LustreError: 358056:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5438.949070] LustreError: 358056:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5438.980074] LustreError: 358056:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5444.071451] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5444.080133] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5444.110708] Lustre: Skipped 1 previous similar message [ 5444.147428] Lustre: Skipped 5 previous similar messages [ 5449.188907] LustreError: 357028:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5449.226572] LustreError: 357028:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5455.840157] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5456.166503] Lustre: server umount lustre-OST0000 complete [ 5464.035435] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5464.047508] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5476.319326] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5476.504829] Lustre: server umount lustre-MDT0000 complete [ 5479.402502] LustreError: 355460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772172753 with bad export cookie 7586349078889919614 [ 5479.406571] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5479.409099] LustreError: 355460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5485.700923] Lustre: server umount lustre-MDT0001 complete [ 5493.563575] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 5495.740954] Key type lgssc unregistered [ 5495.983569] LNet: 359790:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5495.992587] LNetError: 359790:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5496.010328] LNet: Removed LNI 192.168.204.154@tcp [ 5496.610863] Key type .llcrypt unregistered [ 5496.613108] Key type ._llcrypt unregistered [ 5537.571422] Key type ._llcrypt registered [ 5537.573408] Key type .llcrypt registered [ 5537.669827] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 5538.799070] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5538.833988] alg: No test for adler32 (adler32-zlib) [ 5539.824841] Lustre: Lustre: Build Version: 2.17.50_152_g82f9dcf [ 5539.980288] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 5541.615184] Key type lgssc registered [ 5542.419654] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5548.373918] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5548.389722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5549.648794] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5549.659542] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5549.722101] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5552.965311] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 5559.628653] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5559.738645] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5559.793249] Lustre: 362569:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5560.040216] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5563.493762] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 5567.871294] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5570.759903] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5575.650539] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5575.659759] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5575.670974] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5576.681340] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5576.682908] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5578.516209] Lustre: server umount lustre-MDT0000 complete [ 5581.190901] LustreError: 362561:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772172855 with bad export cookie 8410597806340507942 [ 5581.194612] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5581.197058] LustreError: 362561:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5581.417310] Lustre: server umount lustre-MDT0001 complete [ 5587.795871] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 01:14:21 (1772172861) [ 5592.448857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5592.801836] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5595.488359] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 5598.178934] LustreError: 364852:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5598.196654] LustreError: 364852:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5602.021399] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5602.292315] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5605.774931] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 5609.742660] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5612.011453] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5616.563980] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5616.709613] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5616.892619] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5621.347567] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 5624.836881] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5629.764730] Lustre: Permanently deactivating lustre-OST0000 [ 5629.772118] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 5633.842471] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5634.670336] LustreError: 367402:0:(mgs_llog.c:1942:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 5634.675638] LustreError: 367402:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [ 5634.677922] LustreError: 367402:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5636.563531] LustreError: 367450:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [ 5636.567538] LustreError: 367450:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5644.129556] Lustre: server umount lustre-OST0000 complete [ 5645.990480] LustreError: 367700:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [ 5645.995935] LustreError: 367700:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5648.352223] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5648.365758] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5648.378903] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5648.384158] Lustre: Skipped 1 previous similar message [ 5650.402138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5650.405188] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5655.524332] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5655.531318] Lustre: Skipped 2 previous similar messages [ 5657.630688] Lustre: server umount lustre-MDT0000 complete [ 5660.418444] LustreError: 364834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772172934 with bad export cookie 8410597806340508516 [ 5660.421919] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5660.424093] LustreError: 364834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5670.994515] Lustre: server umount lustre-MDT0001 complete [ 5678.143200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5681.759976] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 5683.766198] LustreError: 368841:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 5688.945795] Lustre: 369180:0:(mgs_llog.c:1709:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.154@tcp [ 5690.764470] Lustre: server umount MGS complete [ 5699.159290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5699.563638] LustreError: 369905:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5699.588210] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5699.597178] Lustre: Skipped 1 previous similar message [ 5699.656886] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5702.914955] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 5704.683346] LustreError: 369905:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5709.386251] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5709.596594] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5712.875938] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 5716.756421] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5719.128967] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5720.696874] Lustre: Permanently reactivating lustre-OST0000 [ 5720.721383] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 5720.728636] Lustre: Skipped 2 previous similar messages [ 5744.607166] LustreError: 371365:0:(osp_dev.c:692:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 5750.724107] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5750.954813] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5750.961172] Lustre: Skipped 1 previous similar message [ 5755.089787] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 5758.754695] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5770.720366] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5770.723575] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5770.724697] LustreError: Skipped 1 previous similar message [ 5770.727580] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5770.735964] Lustre: Skipped 1 previous similar message [ 5772.928747] Lustre: server umount lustre-OST0000 complete [ 5781.361214] Lustre: server umount lustre-MDT0000 complete [ 5783.333673] LustreError: 371251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772173057 with bad export cookie 8410597806340509573 [ 5783.335266] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5783.337875] LustreError: 371251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5783.508430] Lustre: server umount lustre-MDT0001 complete [ 5788.379933] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 5789.957033] Key type lgssc unregistered [ 5790.155701] LNet: 373915:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5790.159346] LNetError: 373915:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5790.167577] LNet: Removed LNI 192.168.204.154@tcp [ 5790.593817] Key type .llcrypt unregistered [ 5790.595423] Key type ._llcrypt unregistered [ 5799.330765] Key type ._llcrypt registered [ 5799.332157] Key type .llcrypt registered [ 5799.385632] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 5806.992652] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 5807.331455] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5807.372892] alg: No test for adler32 (adler32-zlib) [ 5808.280627] Lustre: Lustre: Build Version: 2.17.50_152_g82f9dcf [ 5808.400522] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 5809.999145] Key type lgssc registered [ 5810.545378] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5825.484764] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5825.493057] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5826.596888] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5826.617440] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5826.656970] Lustre: lustre-MDT0000: new disk, initializing [ 5826.692879] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5826.705047] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5828.285599] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 5833.380049] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5833.442357] Lustre: 377012:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5833.475474] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5833.478038] Lustre: Skipped 1 previous similar message [ 5833.524095] Lustre: lustre-MDT0001: new disk, initializing [ 5833.564821] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5833.583643] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5833.588326] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5835.157111] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 5838.721690] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5840.333666] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5843.218771] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5843.336955] Lustre: lustre-OST0000: new disk, initializing [ 5843.339390] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5843.376741] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5845.645319] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 5849.096961] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5851.053711] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5853.192244] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5853.196189] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5853.240671] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5854.251083] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 5855.865622] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5855.961879] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5858.273123] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5858.273458] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5858.275312] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5858.277682] LustreError: Skipped 1 previous similar message [ 5858.285297] Lustre: Skipped 1 previous similar message [ 5862.890666] Lustre: server umount lustre-OST0000 complete [ 5864.418777] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5864.422916] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5864.427934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5864.431035] Lustre: Skipped 1 previous similar message [ 5868.512511] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5868.512854] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5868.519639] Lustre: Skipped 1 previous similar message [ 5870.594721] Lustre: server umount lustre-MDT0000 complete [ 5872.360158] LustreError: 377004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772173146 with bad export cookie 403489606451601285 [ 5872.364432] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5872.364765] LustreError: 377004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5872.557046] Lustre: server umount lustre-MDT0001 complete [ 5876.655836] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 5877.372074] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 01:19:11 (1772173151) [ 5880.552180] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5880.792086] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5882.485802] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 5885.921221] LustreError: 381083:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5885.927032] LustreError: 381083:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5886.234593] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5886.404814] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5888.046406] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 5890.442261] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5892.082395] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5895.011071] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5895.162290] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5897.547375] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 5899.737801] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5900.608452] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 5906.694257] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 5910.498887] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5910.500055] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5910.500315] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5910.506996] Lustre: Skipped 1 previous similar message [ 5914.522413] Lustre: server umount lustre-OST0000 complete [ 5917.153311] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5917.156744] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5917.162590] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5917.165594] Lustre: Skipped 1 previous similar message [ 5922.142761] Lustre: server umount lustre-MDT0000 complete [ 5923.660306] LustreError: 381064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772173197 with bad export cookie 403489606451602153 [ 5923.664340] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5923.664524] LustreError: 381064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5923.815955] Lustre: server umount lustre-MDT0001 complete [ 5927.896619] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 5929.093423] Key type lgssc unregistered [ 5929.237745] LNet: 384450:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5929.241109] LNetError: 384450:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5929.250360] LNet: Removed LNI 192.168.204.154@tcp [ 5929.632274] Key type .llcrypt unregistered [ 5929.633543] Key type ._llcrypt unregistered [ 5939.121543] Key type ._llcrypt registered [ 5939.122801] Key type .llcrypt registered [ 5939.169934] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 5939.326395] Key type .llcrypt unregistered [ 5939.328383] Key type ._llcrypt unregistered [ 5947.339636] Key type ._llcrypt registered [ 5947.341431] Key type .llcrypt registered [ 5947.382373] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 5947.697407] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5947.706430] alg: No test for adler32 (adler32-zlib) [ 5948.565667] Lustre: Lustre: Build Version: 2.17.50_152_g82f9dcf [ 5948.650623] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 5950.239167] Key type lgssc registered [ 5950.639469] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5953.559896] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5953.566994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5954.767321] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5956.167574] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 5959.249451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5959.375500] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5961.412362] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 5963.361887] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5964.513084] LustreError: 386623:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5964.519736] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5964.524842] LustreError: 386623:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 5965.639281] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5965.771766] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5967.080643] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 5975.007902] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5975.010104] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5975.015313] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5976.032938] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5976.035459] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5976.035907] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5980.366113] Lustre: server umount lustre-OST0000 complete [ 5984.740566] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 5984.743426] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5984.748413] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5984.751052] Lustre: Skipped 1 previous similar message [ 5987.737306] Lustre: server umount lustre-MDT0000 complete [ 5989.099217] LustreError: 388219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772173262 with bad export cookie 5814187905595244874 [ 5989.101701] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5989.103393] LustreError: 388219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5989.224579] Lustre: server umount lustre-MDT0001 complete [ 5993.038560] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 5994.124952] Key type lgssc unregistered [ 5994.254400] LNet: 389588:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5994.257212] LNetError: 389588:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5994.267339] LNet: Removed LNI 192.168.204.154@tcp [ 5994.587614] Key type .llcrypt unregistered [ 5994.589125] Key type ._llcrypt unregistered [ 6003.964540] Key type ._llcrypt registered [ 6003.965906] Key type .llcrypt registered [ 6004.006524] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 6004.333202] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6004.351766] alg: No test for adler32 (adler32-zlib) [ 6005.221165] Lustre: Lustre: Build Version: 2.17.50_152_g82f9dcf [ 6005.317885] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 6006.903204] Key type lgssc registered [ 6007.320948] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6010.095471] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6010.103910] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6025.567330] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6030.687197] LustreError: 390938:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.154@tcp: failed processing log, type 4: rc = -110 [ 6057.311157] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6059.162370] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6061.033291] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6063.363920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6063.557231] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6063.586978] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 6064.926873] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6068.019315] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6068.168341] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6069.694595] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6071.709907] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6072.967943] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6078.432385] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6078.436797] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6078.442159] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6078.943892] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6078.947572] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6078.953892] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6081.505985] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6081.508257] Lustre: Skipped 1 previous similar message [ 6082.517939] Lustre: server umount lustre-OST0000 complete [ 6086.624724] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6086.625533] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6086.628539] Lustre: Skipped 1 previous similar message [ 6090.009896] Lustre: server umount lustre-MDT0000 complete [ 6091.282215] LustreError: 393130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772173365 with bad export cookie 2840588337753741976 [ 6091.285757] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6091.286366] LustreError: 393130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6091.411937] Lustre: server umount lustre-MDT0001 complete [ 6095.177584] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 6096.200945] Key type lgssc unregistered [ 6096.326535] LNet: 394155:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6096.330443] LNetError: 394155:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6096.344372] LNet: Removed LNI 192.168.204.154@tcp [ 6096.651733] Key type .llcrypt unregistered [ 6096.652930] Key type ._llcrypt unregistered [ 6105.968920] Key type ._llcrypt registered [ 6105.970198] Key type .llcrypt registered [ 6106.006549] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 6106.318615] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6106.343869] alg: No test for adler32 (adler32-zlib) [ 6107.206877] Lustre: Lustre: Build Version: 2.17.50_152_g82f9dcf [ 6107.301492] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 6108.879156] Key type lgssc registered [ 6109.248050] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6111.855615] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6111.863300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6113.021942] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6114.324419] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6117.099889] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6117.244466] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6118.523526] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6120.333911] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6121.549836] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6123.786807] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6123.898842] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6125.710692] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6127.461931] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6137.189636] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 6142.433437] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6142.434515] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6142.437633] Lustre: Skipped 2 previous similar messages [ 6142.439559] Lustre: Skipped 1 previous similar message [ 6143.149668] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.54@tcp (stopping) [ 6143.153358] Lustre: Skipped 1 previous similar message [ 6145.115251] Lustre: server umount lustre-MDT0000 complete [ 6145.864498] LustreError: 395514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.54@tcp arrived at 1772173419 with bad export cookie 8545884104146626652 [ 6145.871300] LustreError: 395514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6146.532429] LustreError: 395514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772173420 with bad export cookie 8545884104146626036 [ 6146.536948] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6147.552150] LustreError: 395533:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6147.552522] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6147.557415] LustreError: 395533:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6147.563144] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6151.583863] LustreError: 395533:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6151.584197] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6151.589036] LustreError: 395533:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6152.653763] Lustre: server umount lustre-OST0000 complete [ 6154.947287] Lustre: server umount lustre-MDT0001 complete [ 6158.881019] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 6159.949646] Key type lgssc unregistered [ 6160.083333] LNet: 398881:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6160.086379] LNetError: 398881:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6160.094300] LNet: Removed LNI 192.168.204.154@tcp [ 6160.372061] Key type .llcrypt unregistered [ 6160.373394] Key type ._llcrypt unregistered [ 6169.542329] Key type ._llcrypt registered [ 6169.543494] Key type .llcrypt registered [ 6169.579445] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 6169.961969] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6169.977605] alg: No test for adler32 (adler32-zlib) [ 6170.828514] Lustre: Lustre: Build Version: 2.17.50_152_g82f9dcf [ 6170.913695] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 6172.495246] Key type lgssc registered [ 6172.876912] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6175.523128] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6175.529466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6176.695509] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6178.009958] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6180.953894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6181.092702] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6182.421720] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6184.305814] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6185.510778] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6187.761349] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6187.875259] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6189.722101] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6191.506920] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6201.191388] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 6206.433876] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6206.434634] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6206.438748] Lustre: Skipped 1 previous similar message [ 6207.149423] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.54@tcp (stopping) [ 6207.151698] Lustre: Skipped 1 previous similar message [ 6211.552202] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6211.554724] Lustre: Skipped 1 previous similar message [ 6216.673994] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6216.676085] Lustre: Skipped 2 previous similar messages [ 6217.183094] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6217.235837] Lustre: server umount lustre-MDT0001 complete [ 6217.380366] LustreError: 400974:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6217.386932] LustreError: 400974:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6221.792702] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6221.793159] LustreError: 400258:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6221.793318] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6221.793968] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6224.979285] Lustre: server umount lustre-OST0000 complete [ 6226.426851] Lustre: server umount lustre-MDT0000 complete [ 6230.880168] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 6231.823896] Key type lgssc unregistered [ 6231.939312] LNet: 403608:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6231.941566] LNetError: 403608:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6231.950375] LNet: Removed LNI 192.168.204.154@tcp [ 6232.231354] Key type .llcrypt unregistered [ 6232.233059] Key type ._llcrypt unregistered [ 6240.067840] Key type ._llcrypt registered [ 6240.068950] Key type .llcrypt registered [ 6240.107971] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 6246.126375] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 6246.471532] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6246.500960] alg: No test for adler32 (adler32-zlib) [ 6247.352110] Lustre: Lustre: Build Version: 2.17.50_152_g82f9dcf [ 6247.433438] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 6249.007137] Key type lgssc registered [ 6249.355500] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6261.215373] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 6266.430083] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6266.436285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6267.526853] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6267.540261] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6267.576293] Lustre: lustre-MDT0000: new disk, initializing [ 6267.602084] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6267.609148] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6268.896358] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6274.735357] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6274.839605] Lustre: lustre-OST0000: new disk, initializing [ 6274.841657] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6274.843966] Lustre: Skipped 1 previous similar message [ 6274.868594] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6276.842334] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6280.109883] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6284.295746] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 6284.299037] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 6284.311840] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 6290.990317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6291.041276] Lustre: 407542:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6291.072841] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6291.112469] Lustre: lustre-MDT0001: new disk, initializing [ 6291.162778] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6291.182568] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 6291.185414] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 6292.601559] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6296.292259] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6296.373603] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6297.746067] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6297.826828] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6301.869646] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.54@tcp (stopping) [ 6303.714332] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6303.716062] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6303.716627] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6303.719377] Lustre: Skipped 1 previous similar message [ 6305.626527] Lustre: server umount lustre-MDT0001 complete [ 6313.953040] LustreError: 407549:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6313.957816] LustreError: 407549:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 6319.072696] LustreError: 407554:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6319.072829] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6319.082687] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6319.084536] Lustre: Skipped 1 previous similar message [ 6324.192233] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6324.192725] LustreError: 407554:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6324.323646] Lustre: server umount lustre-OST0000 complete [ 6332.020980] Lustre: server umount lustre-MDT0000 complete [ 6336.935302] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 6338.138289] Key type lgssc unregistered [ 6338.283846] LNet: 412107:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6338.288340] LNetError: 412107:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6338.308329] LNet: Removed LNI 192.168.204.154@tcp [ 6338.667365] Key type .llcrypt unregistered [ 6338.668729] Key type ._llcrypt unregistered [ 6352.295061] Key type ._llcrypt registered [ 6352.296483] Key type .llcrypt registered [ 6352.337461] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 6352.442505] Key type .llcrypt unregistered [ 6352.444112] Key type ._llcrypt unregistered [ 6359.033707] Key type ._llcrypt registered [ 6359.035160] Key type .llcrypt registered [ 6359.075607] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 6359.509368] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6359.536354] alg: No test for adler32 (adler32-zlib) [ 6360.384589] Lustre: Lustre: Build Version: 2.17.50_152_g82f9dcf [ 6360.459043] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 6362.039167] Key type lgssc registered [ 6362.382650] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6371.522242] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6371.529022] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6372.624767] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6372.635649] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6372.670482] Lustre: lustre-MDT0000: new disk, initializing [ 6372.696082] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6372.703143] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6374.026743] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6378.259329] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6378.302525] Lustre: 415424:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6378.325888] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6378.327917] Lustre: Skipped 1 previous similar message [ 6378.364652] Lustre: lustre-MDT0001: new disk, initializing [ 6378.400290] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6378.415372] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6378.418847] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6379.710046] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6382.836755] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6384.081855] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6386.471560] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6386.577775] Lustre: lustre-OST0000: new disk, initializing [ 6386.579540] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6386.607645] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6387.830915] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6387.834368] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6387.870654] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6388.515615] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6391.787298] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6398.435090] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6398.436597] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6398.439871] Lustre: Skipped 2 previous similar messages [ 6398.443030] Lustre: Skipped 1 previous similar message [ 6400.296115] Lustre: server umount lustre-MDT0000 complete [ 6401.784308] LustreError: 417162:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772173675 with bad export cookie 12723409930342324745 [ 6401.787345] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6401.787569] LustreError: 417162:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6401.908390] Lustre: server umount lustre-MDT0001 complete [ 6413.468519] Lustre: server umount lustre-OST0000 complete [ 6417.208373] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 01:28:11 (1772173691) [ 6420.127357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6420.347808] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6421.765905] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6424.955325] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6425.093450] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6426.469314] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6428.431557] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6429.739663] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6443.230859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6443.321499] Lustre: Found index 0 for lustre-OST0000, updating log [ 6443.398041] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6445.415430] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6446.504661] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 6447.328680] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6449.569120] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6449.569994] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6450.673854] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6450.676128] Lustre: Skipped 1 previous similar message [ 6452.768698] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6452.770927] Lustre: Skipped 2 previous similar messages [ 6454.604676] Lustre: server umount lustre-OST0000 complete [ 6470.112316] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6470.113308] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6470.115582] Lustre: Skipped 1 previous similar message [ 6470.117368] Lustre: Skipped 1 previous similar message [ 6473.364140] Lustre: server umount lustre-MDT0000 complete [ 6474.636463] LustreError: 419058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772173748 with bad export cookie 12723409930342326124 [ 6474.639529] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6474.641025] LustreError: 419058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6474.761846] Lustre: server umount lustre-MDT0001 complete [ 6477.612895] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 01:29:11 (1772173751) [ 6509.983276] LNet: 1 local NIs in recovery (showing 1): 192.168.204.154@tcp [ 7059.516321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7059.694149] LustreError: 425141:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7059.701148] LustreError: 425141:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7059.729358] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7061.202446] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7064.567882] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7066.333548] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7068.499158] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7069.846403] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7083.402240] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7083.499911] Lustre: Found index 0 for lustre-OST0000, updating log [ 7085.723560] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7086.695492] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [ 7087.662858] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7090.465886] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7090.473925] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7090.476080] Lustre: Skipped 1 previous similar message [ 7091.491274] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7094.547410] Lustre: server umount lustre-OST0000 complete [ 7097.057117] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7097.060160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7097.061911] Lustre: Skipped 1 previous similar message [ 7101.155200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7101.158532] Lustre: Skipped 7 previous similar messages [ 7102.167576] Lustre: server umount lustre-MDT0000 complete [ 7102.177746] LustreError: 425854:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7102.183410] LustreError: 425854:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7103.200563] LustreError: 425854:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7103.207535] LustreError: 425854:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7103.590146] LustreError: 425121:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772174377 with bad export cookie 12723409930342327181 [ 7103.593776] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7103.594352] LustreError: 425121:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7103.719186] Lustre: server umount lustre-MDT0001 complete [ 7110.047381] LNet: 1 local NIs in recovery (showing 1): 192.168.204.154@tcp [ 7121.491980] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 7124.862402] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 7137.842729] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 01:40:11 (1772174411) [ 7144.025093] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 7145.134102] Key type lgssc unregistered [ 7145.277638] LNet: 431406:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7145.280352] LNetError: 431406:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7214.949454] LNet: Removed LNI 192.168.204.154@tcp [ 7215.270818] Key type .llcrypt unregistered [ 7215.272166] Key type ._llcrypt unregistered [ 7222.106080] Key type ._llcrypt registered [ 7222.107210] Key type .llcrypt registered [ 7222.143783] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 7222.482223] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7222.531620] alg: No test for adler32 (adler32-zlib) [ 7223.405234] Lustre: Lustre: Build Version: 2.17.50_152_g82f9dcf [ 7223.498529] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 7225.087182] Key type lgssc registered [ 7225.487056] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7227.061736] LNet: Added LNI 192.168.204.154@tcp100 [8/256/0/180] [ 7227.388090] LNet: Added LNI 192.168.204.154@tcp101 [8/256/0/180] [ 7227.688335] LNet: Added LNI 192.168.204.154@tcp102 [8/256/0/180] [ 7228.000538] LNet: Added LNI 192.168.204.154@tcp103 [8/256/0/180] [ 7228.321132] LNet: Added LNI 192.168.204.154@tcp104 [8/256/0/180] [ 7228.644965] LNet: Added LNI 192.168.204.154@tcp105 [8/256/0/180] [ 7228.968660] LNet: Added LNI 192.168.204.154@tcp106 [8/256/0/180] [ 7229.287179] LNet: Added LNI 192.168.204.154@tcp107 [8/256/0/180] [ 7229.606696] LNet: Added LNI 192.168.204.154@tcp108 [8/256/0/180] [ 7229.925140] LNet: Added LNI 192.168.204.154@tcp109 [8/256/0/180] [ 7230.249450] LNet: Added LNI 192.168.204.154@tcp110 [8/256/0/180] [ 7230.544943] LNet: Added LNI 192.168.204.154@tcp111 [8/256/0/180] [ 7230.847718] LNet: Added LNI 192.168.204.154@tcp112 [8/256/0/180] [ 7231.148289] LNet: Added LNI 192.168.204.154@tcp113 [8/256/0/180] [ 7231.454951] LNet: Added LNI 192.168.204.154@tcp114 [8/256/0/180] [ 7231.768080] LNet: Added LNI 192.168.204.154@tcp115 [8/256/0/180] [ 7232.083814] LNet: Added LNI 192.168.204.154@tcp116 [8/256/0/180] [ 7232.401394] LNet: Added LNI 192.168.204.154@tcp117 [8/256/0/180] [ 7232.716337] LNet: Added LNI 192.168.204.154@tcp118 [8/256/0/180] [ 7233.031806] LNet: Added LNI 192.168.204.154@tcp119 [8/256/0/180] [ 7233.343294] LNet: Added LNI 192.168.204.154@tcp120 [8/256/0/180] [ 7233.660236] LNet: Added LNI 192.168.204.154@tcp121 [8/256/0/180] [ 7233.963364] LNet: Added LNI 192.168.204.154@tcp122 [8/256/0/180] [ 7234.262111] LNet: Added LNI 192.168.204.154@tcp123 [8/256/0/180] [ 7234.577591] LNet: Added LNI 192.168.204.154@tcp124 [8/256/0/180] [ 7234.913637] LNet: Added LNI 192.168.204.154@tcp125 [8/256/0/180] [ 7235.246582] LNet: Added LNI 192.168.204.154@tcp126 [8/256/0/180] [ 7235.607266] LNet: Added LNI 192.168.204.154@tcp127 [8/256/0/180] [ 7235.969953] LNet: Added LNI 192.168.204.154@tcp128 [8/256/0/180] [ 7236.314668] LNet: Added LNI 192.168.204.154@tcp129 [8/256/0/180] [ 7236.618900] LNet: Added LNI 192.168.204.154@tcp130 [8/256/0/180] [ 7236.939659] LNet: Added LNI 192.168.204.154@tcp131 [8/256/0/180] [ 7237.247806] LNet: Added LNI 192.168.204.154@tcp132 [8/256/0/180] [ 7237.565196] LNet: Added LNI 192.168.204.154@tcp133 [8/256/0/180] [ 7237.872395] LNet: Added LNI 192.168.204.154@tcp134 [8/256/0/180] [ 7238.182241] LNet: Added LNI 192.168.204.154@tcp135 [8/256/0/180] [ 7240.587379] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7240.596076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7241.650197] Lustre: 434386:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.154@tcp: can't add NID 192.168.204.154@tcp130: rc = -75 [ 7241.721574] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7241.739926] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7241.779811] Lustre: lustre-MDT0000: new disk, initializing [ 7241.834540] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7241.842624] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7243.250812] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7247.477615] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7247.532513] Lustre: 434404:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7247.559438] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7247.561341] Lustre: Skipped 1 previous similar message [ 7247.595689] Lustre: lustre-MDT0001: new disk, initializing [ 7247.611295] Lustre: 435187:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.154@tcp: can't add NID 192.168.204.154@tcp130: rc = -75 [ 7247.614692] Lustre: 435187:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 7247.661190] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7247.676499] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7247.681044] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7249.065353] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7252.338664] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7253.628984] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7255.542224] Lustre: 436042:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.154@tcp: can't add NID 192.168.204.154@tcp130: rc = -75 [ 7255.545123] Lustre: 436042:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 7256.145425] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7256.260578] Lustre: lustre-OST0000: new disk, initializing [ 7256.263544] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7256.314719] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7258.186380] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7261.424578] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7263.228958] Lustre: 436882:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.154@tcp: can't add NID 192.168.204.154@tcp130: rc = -75 [ 7263.234838] Lustre: 436882:0:(lustre_peer.c:105:class_add_uuid()) Skipped 17 previous similar messages [ 7263.259297] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7263.264456] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7263.317204] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7268.323724] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7268.324288] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7268.326262] LustreError: Skipped 1 previous similar message [ 7268.326948] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7268.330789] Lustre: Skipped 1 previous similar message [ 7268.334603] Lustre: Skipped 1 previous similar message [ 7270.055273] Lustre: server umount lustre-OST0000 complete [ 7273.440873] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7273.441113] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7273.443691] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7273.447282] Lustre: Skipped 1 previous similar message [ 7277.592834] Lustre: server umount lustre-MDT0000 complete [ 7278.975174] LustreError: 435186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772174552 with bad export cookie 4960524786605190601 [ 7278.977970] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7278.981596] LustreError: 435186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7279.099617] Lustre: server umount lustre-MDT0001 complete [ 7282.671901] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 7283.657655] Key type lgssc unregistered [ 7283.810647] LNet: 437930:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7283.813514] LNetError: 437930:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7283.816339] LNet: Removed LNI 192.168.204.154@tcp [ 7284.831162] LNet: Removed LNI 192.168.204.154@tcp100 [ 7284.832383] LNetError: 437930:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7284.834044] LNetError: 437930:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 7284.836464] LNet: Removed LNI 192.168.204.154@tcp101 [ 7285.855125] LNet: Removed LNI 192.168.204.154@tcp102 [ 7285.856296] LNetError: 437930:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7285.859320] LNetError: 437930:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 7285.862443] LNet: Removed LNI 192.168.204.154@tcp103 [ 7286.879147] LNet: Removed LNI 192.168.204.154@tcp104 [ 7287.903106] LNet: Removed LNI 192.168.204.154@tcp105 [ 7287.904573] LNetError: 437930:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7287.907870] LNetError: 437930:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 7287.911237] LNet: Removed LNI 192.168.204.154@tcp106 [ 7288.927156] LNet: Removed LNI 192.168.204.154@tcp107 [ 7289.951161] LNet: Removed LNI 192.168.204.154@tcp108 [ 7290.975182] LNet: Removed LNI 192.168.204.154@tcp109 [ 7291.999119] LNet: Removed LNI 192.168.204.154@tcp110 [ 7292.000358] LNetError: 437930:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7292.003162] LNetError: 437930:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 7292.006466] LNet: Removed LNI 192.168.204.154@tcp111 [ 7293.023134] LNet: Removed LNI 192.168.204.154@tcp112 [ 7294.047159] LNet: Removed LNI 192.168.204.154@tcp113 [ 7294.048854] LNet: Removed LNI 192.168.204.154@tcp114 [ 7294.050309] LNet: Removed LNI 192.168.204.154@tcp115 [ 7294.051805] LNet: Removed LNI 192.168.204.154@tcp116 [ 7294.052909] LNet: Removed LNI 192.168.204.154@tcp117 [ 7294.053969] LNet: Removed LNI 192.168.204.154@tcp118 [ 7294.055318] LNet: Removed LNI 192.168.204.154@tcp119 [ 7294.056236] LNet: Removed LNI 192.168.204.154@tcp120 [ 7294.057851] LNet: Removed LNI 192.168.204.154@tcp121 [ 7294.059370] LNet: Removed LNI 192.168.204.154@tcp122 [ 7294.060946] LNet: Removed LNI 192.168.204.154@tcp123 [ 7294.062139] LNet: Removed LNI 192.168.204.154@tcp124 [ 7294.063126] LNet: Removed LNI 192.168.204.154@tcp125 [ 7294.064310] LNet: Removed LNI 192.168.204.154@tcp126 [ 7294.065592] LNet: Removed LNI 192.168.204.154@tcp127 [ 7294.066883] LNet: Removed LNI 192.168.204.154@tcp128 [ 7294.068236] LNet: Removed LNI 192.168.204.154@tcp129 [ 7294.069776] LNet: Removed LNI 192.168.204.154@tcp130 [ 7294.071134] LNet: Removed LNI 192.168.204.154@tcp131 [ 7294.072776] LNet: Removed LNI 192.168.204.154@tcp132 [ 7294.074386] LNet: Removed LNI 192.168.204.154@tcp133 [ 7294.075885] LNet: Removed LNI 192.168.204.154@tcp134 [ 7294.083449] LNet: Removed LNI 192.168.204.154@tcp135 [ 7294.392862] Key type .llcrypt unregistered [ 7294.394131] Key type ._llcrypt unregistered [ 7302.339618] Key type ._llcrypt registered [ 7302.341134] Key type .llcrypt registered [ 7302.386138] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 7308.641448] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 7309.112056] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7309.124731] alg: No test for adler32 (adler32-zlib) [ 7310.002246] Lustre: Lustre: Build Version: 2.17.50_152_g82f9dcf [ 7310.096196] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 7311.679093] Key type lgssc registered [ 7312.062910] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7331.374981] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 01:43:24 (1772174604) [ 7343.478380] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 7345.978967] Key type lgssc unregistered [ 7346.232295] LNet: 441937:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7346.237521] LNetError: 441937:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7346.278685] LNet: Removed LNI 192.168.204.154@tcp [ 7347.241476] Key type .llcrypt unregistered [ 7347.243235] Key type ._llcrypt unregistered [ 7363.479658] Key type ._llcrypt registered [ 7363.481967] Key type .llcrypt registered [ 7363.594751] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 7364.654307] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7364.687692] alg: No test for adler32 (adler32-zlib) [ 7365.747546] Lustre: Lustre: Build Version: 2.17.50_152_g82f9dcf [ 7365.927429] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 7367.567162] Key type lgssc registered [ 7368.206325] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7373.624595] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7373.644883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7374.815866] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7374.837415] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7374.882440] Lustre: lustre-MDT0000: new disk, initializing [ 7374.948884] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7374.962277] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7376.869914] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7383.146766] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7383.214163] Lustre: 443142:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7383.246359] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7383.248425] Lustre: Skipped 1 previous similar message [ 7383.293630] Lustre: lustre-MDT0001: new disk, initializing [ 7383.357304] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7383.380272] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7383.389368] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7385.682531] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7390.103123] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7391.888596] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7394.972251] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7395.095411] Lustre: lustre-OST0000: new disk, initializing [ 7395.098969] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7395.154765] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7397.727543] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7397.734201] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7397.810927] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7398.198268] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7402.325817] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7403.830576] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing lnet_if_list [ 7406.170452] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [ 7406.181499] LNet: Added LNI 192.168.204.154@tcp10 [8/256/0/180] [ 7408.683956] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [ 7408.699723] LNet: Added LNI 192.168.204.154@tcp11 [8/256/0/180] [ 7410.577907] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [ 7410.596174] LNet: Added LNI 192.168.204.154@tcp12 [8/256/0/180] [ 7412.743434] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [ 7412.756657] LNet: Added LNI 192.168.204.154@tcp13 [8/256/0/180] [ 7414.517866] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [ 7414.534553] LNet: Added LNI 192.168.204.154@tcp14 [8/256/0/180] [ 7416.477539] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [ 7416.491388] LNet: Added LNI 192.168.204.154@tcp15 [8/256/0/180] [ 7418.500650] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [ 7418.515633] LNet: Added LNI 192.168.204.154@tcp16 [8/256/0/180] [ 7420.537743] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [ 7420.554151] LNet: Added LNI 192.168.204.154@tcp17 [8/256/0/180] [ 7422.732898] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [ 7422.749569] LNet: Added LNI 192.168.204.154@tcp18 [8/256/0/180] [ 7424.917855] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [ 7424.934473] LNet: Added LNI 192.168.204.154@tcp19 [8/256/0/180] [ 7427.450185] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [ 7427.467562] LNet: Added LNI 192.168.204.154@tcp20 [8/256/0/180] [ 7430.225124] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [ 7430.241963] LNet: Added LNI 192.168.204.154@tcp21 [8/256/0/180] [ 7432.890040] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [ 7432.909969] LNet: Added LNI 192.168.204.154@tcp22 [8/256/0/180] [ 7435.499384] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [ 7435.512733] LNet: Added LNI 192.168.204.154@tcp23 [8/256/0/180] [ 7438.029702] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [ 7438.046590] LNet: Added LNI 192.168.204.154@tcp24 [8/256/0/180] [ 7440.471154] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [ 7440.483746] LNet: Added LNI 192.168.204.154@tcp25 [8/256/0/180] [ 7443.179685] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [ 7443.200984] LNet: Added LNI 192.168.204.154@tcp26 [8/256/0/180] [ 7445.716676] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [ 7445.736197] LNet: Added LNI 192.168.204.154@tcp27 [8/256/0/180] [ 7448.424475] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [ 7448.441281] LNet: Added LNI 192.168.204.154@tcp28 [8/256/0/180] [ 7451.431981] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [ 7451.450532] LNet: Added LNI 192.168.204.154@tcp29 [8/256/0/180] [ 7454.666799] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [ 7454.684059] LNet: Added LNI 192.168.204.154@tcp30 [8/256/0/180] [ 7457.590615] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [ 7457.608349] LNet: Added LNI 192.168.204.154@tcp31 [8/256/0/180] [ 7460.578563] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [ 7460.595276] LNet: Added LNI 192.168.204.154@tcp32 [8/256/0/180] [ 7463.452531] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [ 7463.467230] LNet: Added LNI 192.168.204.154@tcp33 [8/256/0/180] [ 7466.215755] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [ 7466.228768] LNet: Added LNI 192.168.204.154@tcp34 [8/256/0/180] [ 7468.736629] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [ 7468.753502] LNet: Added LNI 192.168.204.154@tcp35 [8/256/0/180] [ 7472.279423] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [ 7472.300205] LNet: Added LNI 192.168.204.154@tcp36 [8/256/0/180] [ 7476.047602] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [ 7476.075370] LNet: Added LNI 192.168.204.154@tcp37 [8/256/0/180] [ 7480.013335] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [ 7480.039127] LNet: Added LNI 192.168.204.154@tcp38 [8/256/0/180] [ 7484.083797] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [ 7484.098920] LNet: Added LNI 192.168.204.154@tcp39 [8/256/0/180] [ 7487.597297] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [ 7487.612557] LNet: Added LNI 192.168.204.154@tcp40 [8/256/0/180] [ 7491.027228] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [ 7491.039633] LNet: Added LNI 192.168.204.154@tcp41 [8/256/0/180] [ 7494.232844] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 7494.251094] LNet: Added LNI 192.168.204.154@tcp42 [8/256/0/180] [ 7498.570953] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [ 7498.594634] LNet: Added LNI 192.168.204.154@tcp43 [8/256/0/180] [ 7503.023312] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [ 7503.042946] LNet: Added LNI 192.168.204.154@tcp44 [8/256/0/180] [ 7507.312950] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [ 7507.336833] LNet: Added LNI 192.168.204.154@tcp45 [8/256/0/180] [ 7511.135421] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [ 7511.154276] LNet: Added LNI 192.168.204.154@tcp46 [8/256/0/180] [ 7514.911724] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [ 7514.931902] LNet: Added LNI 192.168.204.154@tcp47 [8/256/0/180] [ 7519.133865] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [ 7519.151396] LNet: Added LNI 192.168.204.154@tcp48 [8/256/0/180] [ 7523.140434] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [ 7523.159087] LNet: Added LNI 192.168.204.154@tcp49 [8/256/0/180] [ 7526.753694] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [ 7526.768728] LNet: Added LNI 192.168.204.154@tcp50 [8/256/0/180] [ 7539.377377] Lustre: server umount lustre-MDT0000 complete [ 7541.217019] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7541.223055] Lustre: Skipped 2 previous similar messages [ 7541.574916] LustreError: 443136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772174815 with bad export cookie 15839685281933017368 [ 7541.580460] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7541.582606] LustreError: 443136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7541.722801] Lustre: server umount lustre-MDT0001 complete [ 7551.642074] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 7555.569584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7562.722849] Lustre: 442417:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772174820/real 1772174820] req@0000000078fce852 x1858259794366336/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1772174836 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7562.740410] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7566.879558] LustreError: 442413:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000006b0cd4ca x1858259794367232/t0(0) o250->MGC192.168.204.154@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7567.036827] LustreError: 444878:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7567.046520] LustreError: 444878:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 7567.082176] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7567.840226] Lustre: 442414:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772174825/real 1772174825] req@00000000c3c34c0d x1858259794366592/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1772174841 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7569.117196] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7572.193963] LustreError: 468293:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7572.207273] LustreError: 468293:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7573.471192] Lustre: 442414:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772174830/real 1772174830] req@00000000524b5562 x1858259794366720/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1772174846 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7573.978863] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7574.225737] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7576.595352] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7577.311185] Lustre: 442416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772174835/real 1772174835] req@00000000a26369d9 x1858259794366848/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1772174851 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7579.294728] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7579.632456] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7581.241284] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7581.669682] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7591.904791] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7591.914227] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7591.916833] Lustre: Skipped 1 previous similar message [ 7593.784875] Lustre: server umount lustre-OST0000 complete [ 7597.025021] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7597.027738] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7597.029470] Lustre: Skipped 2 previous similar messages [ 7597.037097] Lustre: Skipped 1 previous similar message [ 7601.867647] Lustre: server umount lustre-MDT0000 complete [ 7602.144718] LustreError: 468297:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7602.162155] LustreError: 468297:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7604.116822] LustreError: 443136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772174877 with bad export cookie 15839685281933019601 [ 7604.121991] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7604.122726] LustreError: 443136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7604.284373] Lustre: server umount lustre-MDT0001 complete [ 7609.576656] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 7611.007061] Key type lgssc unregistered [ 7611.234896] LNet: 470829:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7611.237930] LNetError: 470829:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7611.243165] LNet: Removed LNI 192.168.204.154@tcp [ 7611.244917] LNet: Removed LNI 192.168.204.154@tcp10 [ 7611.246438] LNet: Removed LNI 192.168.204.154@tcp11 [ 7611.248348] LNet: Removed LNI 192.168.204.154@tcp12 [ 7611.250846] LNet: Removed LNI 192.168.204.154@tcp13 [ 7611.252860] LNet: Removed LNI 192.168.204.154@tcp14 [ 7611.254873] LNet: Removed LNI 192.168.204.154@tcp15 [ 7611.257369] LNet: Removed LNI 192.168.204.154@tcp16 [ 7611.258748] LNet: Removed LNI 192.168.204.154@tcp17 [ 7611.260661] LNet: Removed LNI 192.168.204.154@tcp18 [ 7611.262642] LNet: Removed LNI 192.168.204.154@tcp19 [ 7611.265438] LNet: Removed LNI 192.168.204.154@tcp20 [ 7611.272295] LNet: Removed LNI 192.168.204.154@tcp21 [ 7611.274536] LNet: Removed LNI 192.168.204.154@tcp22 [ 7611.278105] LNet: Removed LNI 192.168.204.154@tcp23 [ 7611.281100] LNet: Removed LNI 192.168.204.154@tcp24 [ 7611.284981] LNet: Removed LNI 192.168.204.154@tcp25 [ 7611.289492] LNet: Removed LNI 192.168.204.154@tcp26 [ 7611.292105] LNet: Removed LNI 192.168.204.154@tcp27 [ 7611.294662] LNet: Removed LNI 192.168.204.154@tcp28 [ 7611.297901] LNet: Removed LNI 192.168.204.154@tcp29 [ 7611.303045] LNet: Removed LNI 192.168.204.154@tcp30 [ 7611.304956] LNet: Removed LNI 192.168.204.154@tcp31 [ 7611.306200] LNet: Removed LNI 192.168.204.154@tcp32 [ 7611.307930] LNet: Removed LNI 192.168.204.154@tcp33 [ 7611.309540] LNet: Removed LNI 192.168.204.154@tcp34 [ 7611.311192] LNet: Removed LNI 192.168.204.154@tcp35 [ 7611.313862] LNet: Removed LNI 192.168.204.154@tcp36 [ 7611.315503] LNet: Removed LNI 192.168.204.154@tcp37 [ 7611.317825] LNet: Removed LNI 192.168.204.154@tcp38 [ 7611.319142] LNet: Removed LNI 192.168.204.154@tcp39 [ 7611.320616] LNet: Removed LNI 192.168.204.154@tcp40 [ 7611.322683] LNet: Removed LNI 192.168.204.154@tcp41 [ 7611.324713] LNet: Removed LNI 192.168.204.154@tcp42 [ 7611.327224] LNet: Removed LNI 192.168.204.154@tcp43 [ 7611.329492] LNet: Removed LNI 192.168.204.154@tcp44 [ 7611.331675] LNet: Removed LNI 192.168.204.154@tcp45 [ 7611.336761] LNet: Removed LNI 192.168.204.154@tcp46 [ 7611.339325] LNet: Removed LNI 192.168.204.154@tcp47 [ 7611.341513] LNet: Removed LNI 192.168.204.154@tcp48 [ 7611.346053] LNet: Removed LNI 192.168.204.154@tcp49 [ 7611.354969] LNet: Removed LNI 192.168.204.154@tcp50 [ 7611.795532] Key type .llcrypt unregistered [ 7611.797155] Key type ._llcrypt unregistered [ 7622.176750] Key type ._llcrypt registered [ 7622.178393] Key type .llcrypt registered [ 7622.243500] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 7622.899128] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7622.919120] alg: No test for adler32 (adler32-zlib) [ 7623.821647] Lustre: Lustre: Build Version: 2.17.50_152_g82f9dcf [ 7623.932099] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 7625.543199] Key type lgssc registered [ 7626.160585] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7630.377086] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7630.389264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7631.665488] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7633.639928] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7638.435357] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7638.650413] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7640.990249] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7644.049405] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7646.136054] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7649.828382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7650.034882] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7653.066321] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7655.778142] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7657.196667] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 7671.900262] loop0: detected capacity change from 0 to 204800000 [ 7671.969613] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7671.973730] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7671.978681] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7674.686791] loop0: detected capacity change from 0 to 204800000 [ 7674.763687] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7674.770754] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7674.777843] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7677.216942] loop0: detected capacity change from 0 to 204800000 [ 7679.784980] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7679.854239] Lustre: fs15246-OST0000: new disk, initializing [ 7679.856996] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 7679.902542] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7682.524795] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7687.786698] loop1: detected capacity change from 0 to 204800000 [ 7690.811896] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7690.893186] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 7690.936175] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 7690.993421] Lustre: fs15246-MDT0000: new disk, initializing [ 7691.072680] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7691.089394] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7691.159811] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 7692.196143] Lustre: 476436:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@000000001ab277e4 x1858260064834176/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1772174982 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 7693.237855] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7701.290510] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 7701.294781] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 7701.307083] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 7705.573859] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7705.581696] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 7709.668640] Lustre: server umount fs15246-MDT0000 complete [ 7710.692090] LustreError: 472079:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7710.699537] LustreError: 472079:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7714.209561] Lustre: server umount fs15246-OST0000 complete [ 7719.906377] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7719.914624] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7719.921033] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7720.929801] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7720.930450] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7726.052325] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7726.056342] Lustre: Skipped 1 previous similar message [ 7731.169366] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7731.174444] Lustre: Skipped 2 previous similar messages [ 7734.239119] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7734.343438] Lustre: server umount lustre-OST0000 complete [ 7740.897121] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7740.901668] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7740.907073] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7742.578110] Lustre: server umount lustre-MDT0000 complete [ 7744.676433] LustreError: 476430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772175018 with bad export cookie 17567450433596756108 [ 7744.680058] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7744.681741] LustreError: 476430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7744.898199] Lustre: server umount lustre-MDT0001 complete [ 7751.203415] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 7752.626381] Key type lgssc unregistered [ 7752.823826] LNet: 479498:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7752.829988] LNetError: 479498:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7752.841738] LNet: Removed LNI 192.168.204.154@tcp [ 7753.265649] Key type .llcrypt unregistered [ 7753.267442] Key type ._llcrypt unregistered [ 7763.401569] Key type ._llcrypt registered [ 7763.404030] Key type .llcrypt registered [ 7763.458164] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 7763.999834] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7764.041837] alg: No test for adler32 (adler32-zlib) [ 7764.973657] Lustre: Lustre: Build Version: 2.17.50_152_g82f9dcf [ 7765.101790] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 7766.703166] Key type lgssc registered [ 7767.259996] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7786.412278] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 7791.337350] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 7809.435964] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7809.449655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7810.586923] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7810.604831] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7810.649886] Lustre: lustre-MDT0000: new disk, initializing [ 7810.698553] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7810.709517] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7812.688602] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7818.186857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7818.267668] Lustre: 483884:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7818.317738] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7818.326247] Lustre: Skipped 1 previous similar message [ 7818.378790] Lustre: lustre-MDT0001: new disk, initializing [ 7818.471894] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7818.503213] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7818.513358] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7820.954561] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7825.541872] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7827.349422] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7830.292559] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7830.415926] Lustre: lustre-OST0000: new disk, initializing [ 7830.418635] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7830.465572] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7832.980821] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7833.424803] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7833.429577] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7833.490878] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7836.819315] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7838.548661] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7838.650265] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7840.441424] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7840.539416] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7843.811659] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7843.812040] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7843.814239] LustreError: Skipped 1 previous similar message [ 7843.819135] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7843.823199] Lustre: Skipped 1 previous similar message [ 7847.531920] Lustre: server umount lustre-OST0000 complete [ 7849.440405] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7849.443497] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7849.448784] Lustre: Skipped 1 previous similar message [ 7849.451486] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7854.050219] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7854.050349] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7854.058381] Lustre: Skipped 1 previous similar message [ 7855.612476] Lustre: server umount lustre-MDT0000 complete [ 7857.800993] LustreError: 483875:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772175131 with bad export cookie 2574010188606222762 [ 7857.802090] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7857.809923] LustreError: 483875:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7857.982691] Lustre: server umount lustre-MDT0001 complete [ 7870.740585] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 7876.469863] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 7893.570906] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7762 sec ========== 01:52:47 (1772175167) [ 7894.439279] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 01:52:48 (1772175168) === [ 7896.156250] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 01:52:49 (1772175169) === [ 7914.528069] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 7916.075831] Key type lgssc unregistered [ 7916.299336] LNet: 492060:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7916.305182] LNetError: 492060:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7916.318709] LNet: Removed LNI 192.168.204.154@tcp [ 7916.846456] Key type .llcrypt unregistered [ 7916.848432] Key type ._llcrypt unregistered