[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 836992794 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2448MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002332] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004017] kvm-guest: setup PV IPIs [ 0.007584] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009013] pid_max: default: 32768 minimum: 301 [ 0.011095] LSM: Security Framework initializing [ 0.012058] Yama: becoming mindful. [ 0.013044] SELinux: Initializing. [ 0.014088] *** VALIDATE selinux *** [ 0.023478] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028359] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030100] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031109] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032114] *** VALIDATE tmpfs *** [ 0.034213] *** VALIDATE proc *** [ 0.035209] *** VALIDATE cgroup *** [ 0.037007] *** VALIDATE cgroup2 *** [ 0.038265] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039154] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041031] Spectre V2 : User space: Vulnerable [ 0.042010] Speculative Store Bypass: Vulnerable [ 0.045613] debug: unmapping init [mem 0xffffffffb4059000-0xffffffffb4060fff] [ 0.047278] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048774] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049025] ... version: 2 [ 0.050011] ... bit width: 48 [ 0.051016] ... generic registers: 4 [ 0.052016] ... value mask: 0000ffffffffffff [ 0.053017] ... max period: 00007fffffffffff [ 0.054016] ... fixed-purpose events: 3 [ 0.055015] ... event mask: 000000070000000f [ 0.056378] rcu: Hierarchical SRCU implementation. [ 0.059073] smp: Bringing up secondary CPUs ... [ 0.060848] x86: Booting SMP configuration: [ 0.061038] .... node #0, CPUs: #1 #2 #3 [ 0.071150] smp: Brought up 1 node, 4 CPUs [ 0.073018] smpboot: Max logical packages: 1 [ 0.074015] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.157853] node 0 deferred pages initialised in 81ms [ 0.161212] devtmpfs: initialized [ 0.163297] x86/mm: Memory block size: 128MB [ 0.167045] gcov: version magic: 0x41383552 [ 0.168996] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.184100] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.186576] pinctrl core: initialized pinctrl subsystem [ 0.189225] [ 0.190009] ************************************************************* [ 0.193017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.196015] ** ** [ 0.200017] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.203015] ** ** [ 0.206016] ** This means that this kernel is built to expose internal ** [ 0.212023] ** IOMMU data structures, which may compromise security on ** [ 0.215015] ** your system. ** [ 0.219012] ** ** [ 0.223015] ** If you see this message and you are not debugging the ** [ 0.226013] ** kernel, report this immediately to your vendor! ** [ 0.230015] ** ** [ 0.234013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.238015] ************************************************************* [ 0.244491] NET: Registered protocol family 16 [ 0.247420] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.251065] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.256171] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.261093] cpuidle: using governor menu [ 0.265425] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.269456] PCI: Using configuration type 1 for base access [ 0.272131] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.282263] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.283014] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.288158] cryptd: max_cpu_qlen set to 1000 [ 0.289385] ACPI: Added _OSI(Module Device) [ 0.291016] ACPI: Added _OSI(Processor Device) [ 0.294015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.297015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.305781] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.317709] ACPI: Interpreter enabled [ 0.320076] ACPI: PM: (supports S0 S3 S4 S5) [ 0.323014] ACPI: Using IOAPIC for interrupt routing [ 0.327141] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.333764] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.356013] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.359040] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.363019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.367069] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.376238] acpiphp: Slot [2] registered [ 0.379187] acpiphp: Slot [5] registered [ 0.381157] acpiphp: Slot [6] registered [ 0.383163] acpiphp: Slot [7] registered [ 0.384125] acpiphp: Slot [8] registered [ 0.386102] acpiphp: Slot [9] registered [ 0.387176] acpiphp: Slot [10] registered [ 0.389095] acpiphp: Slot [3] registered [ 0.390122] acpiphp: Slot [4] registered [ 0.391149] acpiphp: Slot [11] registered [ 0.392165] acpiphp: Slot [12] registered [ 0.394099] acpiphp: Slot [13] registered [ 0.395183] acpiphp: Slot [14] registered [ 0.397084] acpiphp: Slot [15] registered [ 0.398100] acpiphp: Slot [16] registered [ 0.400107] acpiphp: Slot [17] registered [ 0.401081] acpiphp: Slot [18] registered [ 0.402147] acpiphp: Slot [19] registered [ 0.404096] acpiphp: Slot [20] registered [ 0.405076] acpiphp: Slot [21] registered [ 0.406079] acpiphp: Slot [22] registered [ 0.407120] acpiphp: Slot [23] registered [ 0.409093] acpiphp: Slot [24] registered [ 0.410132] acpiphp: Slot [25] registered [ 0.411148] acpiphp: Slot [26] registered [ 0.413141] acpiphp: Slot [27] registered [ 0.414079] acpiphp: Slot [28] registered [ 0.415188] acpiphp: Slot [29] registered [ 0.417158] acpiphp: Slot [30] registered [ 0.418143] acpiphp: Slot [31] registered [ 0.420078] PCI host bridge to bus 0000:00 [ 0.421019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.424023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.426021] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.430025] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.433022] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.437026] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.439290] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.443268] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.447500] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.459840] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.466024] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.469018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.471016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.475018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.477639] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.482267] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.485042] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.487930] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.493013] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.505020] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.509921] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.517074] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.526015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.534020] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.555013] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.565512] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.566000] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.568000] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.597018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.607669] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.631025] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.653026] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.701036] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.719763] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.730015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.744020] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.762022] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.774676] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.786022] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.793023] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.818023] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.832512] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.840021] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.849022] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.868021] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.881000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.883443] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.886454] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.889460] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.892296] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.898214] iommu: Default domain type: Passthrough [ 0.902000] SCSI subsystem initialized [ 0.904250] ACPI: bus type USB registered [ 0.907191] usbcore: registered new interface driver usbfs [ 0.910151] usbcore: registered new interface driver hub [ 0.913115] usbcore: registered new device driver usb [ 0.915279] pps_core: LinuxPPS API ver. 1 registered [ 0.917016] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.921116] PTP clock support registered [ 0.924159] EDAC MC: Ver: 3.0.0 [ 0.927199] PCI: Using ACPI for IRQ routing [ 0.930193] NetLabel: Initializing [ 0.932016] NetLabel: domain hash size = 128 [ 0.934018] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.937170] NetLabel: unlabeled traffic allowed by default [ 0.940424] vgaarb: loaded [ 0.942747] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.945018] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.952227] clocksource: Switched to clocksource kvm-clock [ 1.083081] VFS: Disk quotas dquot_6.6.0 [ 1.084982] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.088074] *** VALIDATE ramfs *** [ 1.089485] *** VALIDATE hugetlbfs *** [ 1.091441] pnp: PnP ACPI init [ 1.094323] pnp: PnP ACPI: found 6 devices [ 1.111870] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.115550] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.118209] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.120800] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.123658] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.126522] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.129679] NET: Registered protocol family 2 [ 1.132871] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.138847] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.143333] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.149405] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.153340] TCP: Hash tables configured (established 65536 bind 65536) [ 1.156664] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.160592] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.164301] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.167987] NET: Registered protocol family 1 [ 1.171679] RPC: Registered named UNIX socket transport module. [ 1.173957] RPC: Registered udp transport module. [ 1.176330] RPC: Registered tcp transport module. [ 1.178302] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.182134] NET: Registered protocol family 44 [ 1.185841] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.189811] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.195885] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.200172] PCI: CLS 0 bytes, default 64 [ 1.203336] Unpacking initramfs... [ 3.934776] debug: unmapping init [mem 0xffffa0c83cc54000-0xffffa0c83ffbffff] [ 3.946188] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.949064] software IO TLB: mapped [mem 0x00000000b8c54000-0x00000000bcc54000] (64MB) [ 3.953601] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 4.905983] Initialise system trusted keyrings [ 4.908035] Key type blacklist registered [ 4.911175] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 4.924479] zbud: loaded [ 4.930399] *** VALIDATE nfs *** [ 4.931786] *** VALIDATE nfs4 *** [ 4.933338] pstore: using deflate compression [ 4.937977] Platform Keyring initialized [ 5.101996] NET: Registered protocol family 38 [ 5.104350] Key type asymmetric registered [ 5.105789] Asymmetric key parser 'x509' registered [ 5.108333] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 5.112344] io scheduler mq-deadline registered [ 5.114564] io scheduler kyber registered [ 5.116889] io scheduler bfq registered [ 5.119187] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 5.122902] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 5.126613] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 5.130519] ACPI: Power Button [PWRF] [ 5.136985] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 5.144819] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 5.161160] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 5.170266] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 5.188622] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 5.218829] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 5.249844] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 5.255447] Non-volatile memory driver v1.3 [ 5.257420] Linux agpgart interface v0.103 [ 5.290330] virtio_blk virtio1: [vda] 145904 512-byte logical blocks (74.7 MB/71.2 MiB) [ 5.293735] vda: detected capacity change from 0 to 74702848 [ 5.313462] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 5.317033] vdb: detected capacity change from 0 to 1073741824 [ 5.343251] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.347025] vdc: detected capacity change from 0 to 2621440000 [ 5.375371] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.379064] vdd: detected capacity change from 0 to 2621440000 [ 5.460927] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.463315] vde: detected capacity change from 0 to 4294967296 [ 5.490425] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.493976] vdf: detected capacity change from 0 to 4294967296 [ 5.504110] libphy: Fixed MDIO Bus: probed [ 5.512911] usbcore: registered new interface driver usbserial_generic [ 5.517370] usbserial: USB Serial support registered for generic [ 5.521183] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 5.539364] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 5.541526] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 5.544830] mousedev: PS/2 mouse device common for all mice [ 5.548221] rtc_cmos 00:05: RTC can wake from S4 [ 5.552893] rtc_cmos 00:05: registered as rtc0 [ 5.555467] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 5.565366] intel_pstate: CPU model not supported [ 5.572042] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 5.586600] hid: raw HID events driver (C) Jiri Kosina [ 5.598517] usbcore: registered new interface driver usbhid [ 5.608962] usbhid: USB HID core driver [ 5.615229] drop_monitor: Initializing network drop monitor service [ 5.628104] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 5.638880] Initializing XFRM netlink socket [ 5.648597] NET: Registered protocol family 10 [ 5.654994] Segment Routing with IPv6 [ 5.656917] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 5.657770] NET: Registered protocol family 17 [ 5.667573] mpls_gso: MPLS GSO support [ 5.672780] RAS: Correctable Errors collector initialized. [ 5.675348] AVX version of gcm_enc/dec engaged. [ 5.677436] AES CTR mode by8 optimization enabled [ 5.786261] sched_clock: Marking stable (5786224491, 0)->(6980204857, -1193980366) [ 5.791977] registered taskstats version 1 [ 5.795774] Loading compiled-in X.509 certificates [ 5.799277] zswap: loaded using pool lzo/zbud [ 5.857234] Key type big_key registered [ 5.888450] Key type encrypted registered [ 5.893977] ima: No TPM chip found, activating TPM-bypass! [ 5.899524] ima: Allocated hash algorithm: sha1 [ 5.904591] ima: No architecture policies found [ 5.908828] evm: Initialising EVM extended attributes: [ 5.913840] evm: security.selinux [ 5.917370] evm: security.ima [ 5.919962] evm: security.capability [ 5.922964] evm: HMAC attrs: 0x1 [ 5.926949] rtc_cmos 00:05: setting system clock to 2026-08-10 05:06:56 UTC (1786338416) [ 5.936829] debug: unmapping init [mem 0xffffffffb5003000-0xffffffffb51fffff] [ 5.941740] debug: unmapping init [mem 0xffffffffb3d82000-0xffffffffb4058fff] [ 5.950722] Write protecting the kernel read-only data: 28672k [ 5.959222] debug: unmapping init [mem 0xffffffffb2403000-0xffffffffb25fffff] [ 5.965827] debug: unmapping init [mem 0xffffffffb2d14000-0xffffffffb2dfffff] [ 6.058294] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 6.067712] systemd[1]: Detected virtualization kvm. [ 6.070372] systemd[1]: Detected architecture x86-64. [ 6.072753] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 6.111858] systemd[1]: No hostname configured. [ 6.113950] systemd[1]: Set hostname to . [ 6.116640] random: systemd: uninitialized urandom read (16 bytes read) [ 6.119931] systemd[1]: Initializing machine ID from random generator. [ 6.682502] random: systemd: uninitialized urandom read (16 bytes read) [ 6.690290] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 6.700819] random: systemd: uninitialized urandom read (16 bytes read) [ 6.705122] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 6.729751] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Swap. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Slices. Starting Journal Service... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Control Socket. Starting Setup Virtual Console... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Sockets. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 8.510634] device-mapper: uevent: version 1.0.3 [ 8.513792] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System.[ 10.172767] random: fast init done [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 10.374218] virtio_net virtio0 ens2: renamed from eth0 [ 10.822972] scsi host0: ata_piix [ 10.911472] scsi host1: ata_piix [ 10.912979] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 10.915755] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 15.505944] random: crng init done [ 15.515290] random: 7 urandom warning(s) missed due to ratelimiting [ 18.234866] dracut-initqueue[582]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ 19.879052] hrtimer: interrupt took 4059743 ns [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 20.138493] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 23.424187] printk: systemd: 26 output lines suppressed due to ratelimiting [ 23.979211] SELinux: Disabled at runtime. [ 24.050648] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 24.062034] systemd[1]: Detected virtualization kvm. [ 24.066690] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 25.949873] systemd[1]: initrd-switch-root.service: Succeeded. [ 25.953600] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 25.967715] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 25.975622] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 25.979760] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 25.989770] systemd[1]: Starting Journal Service... Starting Journal Service... [ 25.996817] systemd[1]: proc-sys-fs-binfmt_misc.automount: Refusing to start, unit to trigger not loaded. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target rpc_pipefs.target. Mounting POSIX Message Queue File System... Starting Remount Root and Kernel File Systems... [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting Huge Pages File System... [ OK ] Listening on udev Control Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Kernel Debug File System... [ 26.478216] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-getty.slice. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. Starting Apply Kernel Variables... Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 27.670206] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 28.769945] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 28.800976] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 30.299268] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 30.371800] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (8s / no limit)[ 35.260947] Key type dns_resolver registered [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ ***] A start job is running for Configur…only root support (10s / no limit) [ **] A start job is running for Configur…only root support (10s / no limit)[ 36.459468] NFS: Registering the id_resolver key type [ 36.465639] Key type id_resolver registered [ 36.467692] Key type id_legacy registered [ *] A start job is running for Configur…only root support (11s / no limit) [ **] A start job is running for Configur…only root support (11s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... Starting Network Manager... Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Hostname Service... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started OpenSSH server daemon. [ OK ] Started Login Service. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg323-server login: [ 89.549829] libcfs: loading out-of-tree module taints kernel. [ 89.584539] Key type ._llcrypt registered [ 89.586538] Key type .llcrypt registered [ 89.649325] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 105.476760] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 107.536388] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 107.549191] alg: No test for adler32 (adler32-zlib) [ 108.800821] Lustre: Lustre: Build Version: 2.17.56_50_g8cc021f [ 109.814702] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 111.581993] Key type lgssc registered [ 113.614126] Lustre: Echo OBD driver; http://www.lustre.org/ [ 130.790623] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 165.047368] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 177.312594] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 177.347037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 178.605407] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 178.623567] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 178.688291] Lustre: lustre-MDT0000: new disk, initializing [ 178.740338] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 178.751383] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 182.832825] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 194.973883] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 195.052460] Lustre: 6511:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 195.098158] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 195.105990] Lustre: Skipped 1 previous similar message [ 195.217678] Lustre: lustre-MDT0001: new disk, initializing [ 195.299968] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 195.363851] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 195.388379] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 199.605950] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 203.695804] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 211.672536] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 211.846189] Lustre: lustre-OST0000: new disk, initializing [ 211.849557] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 211.855902] Lustre: 8447:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 211.900138] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 218.135305] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 218.151440] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 218.175452] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 218.257433] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 232.658661] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 232.765974] Lustre: lustre-OST0001: new disk, initializing [ 232.768919] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 232.773740] Lustre: 9519:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 232.831741] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 238.521355] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 241.781459] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 241.795315] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 241.891436] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 250.427882] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 256.884416] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 263.486513] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing check_logdir /tmp/testlogs/ [ 268.086872] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing yml_node [ 272.827578] Lustre: DEBUG MARKER: Client: 2.17.56.50 [ 274.457375] Lustre: DEBUG MARKER: MDS: 2.17.56.50 [ 276.336783] Lustre: DEBUG MARKER: OSS: 2.17.56.50 [ 277.325701] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Aug 10 01:11:27 EDT 2026 [ 291.922486] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 293.470449] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 308.192412] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 308.200457] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 308.208391] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 308.708405] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 308.709619] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 308.716418] Lustre: Skipped 2 previous similar messages [ 308.727180] Lustre: Skipped 2 previous similar messages [ 313.824459] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 313.829329] Lustre: Skipped 3 previous similar messages [ 314.102833] Lustre: server umount lustre-MDT0000 complete [ 317.032525] LustreError: 6501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786338727 with bad export cookie 13318030350320910957 [ 317.041690] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 317.051188] LustreError: 6501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 318.945363] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 318.948486] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 318.975266] Lustre: Skipped 1 previous similar message [ 318.984040] Lustre: Skipped 1 previous similar message [ 322.980162] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 322.984633] Lustre: Skipped 1 previous similar message [ 323.541068] Lustre: server umount lustre-MDT0001 complete [ 326.442507] Lustre: server umount lustre-OST0000 complete [ 329.576522] Lustre: server umount lustre-OST0001 complete [ 335.525214] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 344.112406] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 376.430935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 376.624965] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 376.654772] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 376.719635] Lustre: lustre-MDT0000: new disk, initializing [ 376.779080] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 376.790713] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 380.712723] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 391.114025] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 391.356332] Lustre: 15070:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 391.403789] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 391.406885] Lustre: Skipped 1 previous similar message [ 391.561857] Lustre: lustre-MDT0001: new disk, initializing [ 391.735114] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 391.788779] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 391.813135] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 395.665681] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 404.029464] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 408.887380] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 415.400552] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 415.562148] Lustre: lustre-OST0000: new disk, initializing [ 415.567184] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 415.604575] Lustre: 17038:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 415.674566] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 417.303702] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 417.328535] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 417.484403] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 421.386955] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 429.421586] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 434.383628] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 434.890478] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 439.245320] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 439.577570] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 441.316459] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 441.325222] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 447.512850] Lustre: server umount lustre-OST0000 complete [ 451.555203] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 451.578115] Lustre: Skipped 1 previous similar message [ 457.728157] Lustre: server umount lustre-MDT0000 complete [ 460.901763] LustreError: 15064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786338871 with bad export cookie 13318030350320912483 [ 460.918131] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 461.154390] Lustre: server umount lustre-MDT0001 complete [ 465.480809] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 01:14:35 (1786338875) [ 467.702933] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 469.502465] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 01:14:39 (1786338879) [ 475.460661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 475.726615] LustreError: 19446:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 475.746708] LustreError: 19446:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 475.777516] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 479.308457] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 481.256890] LustreError: 19447:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 486.371016] LustreError: 19446:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 486.679620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 490.720486] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 497.767465] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 502.137698] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 509.299680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 515.019508] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 520.879955] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 531.429530] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 531.441664] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 531.448037] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 531.448045] Lustre: Skipped 7 previous similar messages [ 531.449230] Lustre: Skipped 2 previous similar messages [ 533.662765] Lustre: server umount lustre-OST0000 complete [ 538.109963] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 551.397203] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 551.518265] Lustre: server umount lustre-MDT0000 complete [ 551.910045] LustreError: 20879:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 551.926515] LustreError: 20879:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 554.986129] LustreError: 19428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786338965 with bad export cookie 13318030350320913372 [ 554.995805] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 554.998561] LustreError: 19428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 555.227774] Lustre: server umount lustre-MDT0001 complete [ 563.868171] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 565.996110] Key type lgssc unregistered [ 566.151687] LNet: 22980:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 566.157097] LNetError: 22980:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 566.169606] LNet: Removed LNI 192.168.203.123@tcp [ 566.825359] Key type .llcrypt unregistered [ 566.827234] Key type ._llcrypt unregistered [ 587.093156] Key type ._llcrypt registered [ 587.095347] Key type .llcrypt registered [ 587.175517] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 588.247562] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 588.310089] alg: No test for adler32 (adler32-zlib) [ 589.279190] Lustre: Lustre: Build Version: 2.17.56_50_g8cc021f [ 589.532242] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 591.183175] Key type lgssc registered [ 592.295557] Lustre: Echo OBD driver; http://www.lustre.org/ [ 600.109061] loop: module loaded [ 600.229116] loop0: detected capacity change from 0 to 204800000 [ 600.483348] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 600.499278] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 600.835608] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 605.858617] loop0: detected capacity change from 0 to 204800000 [ 605.973667] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 605.984287] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 605.993385] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 610.593897] loop0: detected capacity change from 0 to 204800000 [ 616.755712] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 616.773028] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 617.961890] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 618.002544] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 618.086917] Lustre: lustre-MDT0000: new disk, initializing [ 618.209573] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 618.232643] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 622.716357] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 631.928095] loop1: detected capacity change from 0 to 204800000 [ 637.287820] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 637.477957] Lustre: lustre-OST07c6: new disk, initializing [ 637.480941] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 637.484459] Lustre: Skipped 1 previous similar message [ 637.488356] Lustre: 26163:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 637.552436] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 643.322631] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 645.224546] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 645.233590] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 645.328096] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 648.574799] Lustre: 26171:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 255 < left 264, rollback = 2 [ 648.580133] Lustre: 26171:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 648.584489] Lustre: 26171:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 648.589534] Lustre: 26171:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 648.594253] Lustre: 26171:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 648.597987] Lustre: 26171:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 664.285465] Lustre: 25149:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 664.294559] Lustre: 25149:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 664.298983] Lustre: 25149:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 664.303659] Lustre: 25149:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 664.309326] Lustre: 25149:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 664.313984] Lustre: 25149:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 668.631393] Lustre: 25149:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 668.644868] Lustre: 25149:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 668.654343] Lustre: 25149:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 668.666530] Lustre: 25149:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 668.676940] Lustre: 25149:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 668.686416] Lustre: 25149:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 668.692804] Lustre: 25149:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 668.706713] Lustre: 25149:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 668.724823] Lustre: 25149:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 668.732383] Lustre: 25149:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 668.745653] Lustre: 25149:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 668.758123] Lustre: 25149:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 671.918948] Lustre: 25151:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 671.925555] Lustre: 25151:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 671.930791] Lustre: 25151:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 671.935674] Lustre: 25151:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 671.940407] Lustre: 25151:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 671.949782] Lustre: 25151:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 671.955667] Lustre: 25151:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 671.963855] Lustre: 25151:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 671.967140] Lustre: 25151:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 671.970237] Lustre: 25151:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 671.973583] Lustre: 25151:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 671.976923] Lustre: 25151:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 676.158068] Lustre: 26171:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 676.171994] Lustre: 26171:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 676.186333] Lustre: 26171:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 676.195090] Lustre: 26171:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 676.200456] Lustre: 26171:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 676.206106] Lustre: 26171:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 676.211699] Lustre: 26171:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 676.217432] Lustre: 26171:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 676.222724] Lustre: 26171:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 676.228243] Lustre: 26171:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 676.233954] Lustre: 26171:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 676.239819] Lustre: 26171:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 686.700722] Lustre: 25150:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 258 < left 278, rollback = 2 [ 686.709155] Lustre: 25150:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 686.715785] Lustre: 25150:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 686.723881] Lustre: 25150:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 686.734128] Lustre: 25150:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 686.744096] Lustre: 25150:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 686.750073] Lustre: 25150:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 686.755711] Lustre: 25150:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 686.764050] Lustre: 25150:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 686.769416] Lustre: 25150:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 686.775616] Lustre: 25150:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 686.781937] Lustre: 25150:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 737.250114] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 737.256894] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 737.276440] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 738.854873] Lustre: server umount lustre-OST07c6 complete [ 747.651897] Lustre: server umount lustre-MDT0000 complete [ 761.985949] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 764.630313] Key type lgssc unregistered [ 764.970719] LNet: 28726:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 764.983384] LNetError: 28726:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 765.003582] LNet: Removed LNI 192.168.203.123@tcp [ 765.716187] Key type .llcrypt unregistered [ 765.718232] Key type ._llcrypt unregistered [ 780.623811] Key type ._llcrypt registered [ 780.625872] Key type .llcrypt registered [ 780.701980] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 781.671700] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 781.738090] alg: No test for adler32 (adler32-zlib) [ 782.799227] Lustre: Lustre: Build Version: 2.17.56_50_g8cc021f [ 783.078226] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 784.756148] Key type lgssc registered [ 785.553679] Lustre: Echo OBD driver; http://www.lustre.org/ [ 792.478787] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 792.508931] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 793.832840] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 797.542976] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 804.323214] LustreError: 29976:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 804.350414] LustreError: 29976:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 804.703232] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 805.042307] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 808.751954] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 816.354860] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 822.211618] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 828.821747] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 829.046239] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 835.715185] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 839.283497] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 842.177375] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 855.009468] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 855.023943] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 855.034602] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 855.038658] Lustre: Skipped 1 previous similar message [ 860.131100] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 860.138216] Lustre: Skipped 1 previous similar message [ 861.109275] Lustre: server umount lustre-OST0000 complete [ 865.252664] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 865.258697] Lustre: Skipped 1 previous similar message [ 865.261618] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 870.369182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 870.381528] Lustre: Skipped 2 previous similar messages [ 870.957861] Lustre: server umount lustre-MDT0000 complete [ 874.385793] LustreError: 29955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786339284 with bad export cookie 8563411271435446124 [ 874.388350] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 874.397066] LustreError: 29955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 874.652233] Lustre: server umount lustre-MDT0001 complete [ 884.410846] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 886.460592] Key type lgssc unregistered [ 886.705557] LNet: 33715:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 886.716413] LNetError: 33715:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 886.739896] LNet: Removed LNI 192.168.203.123@tcp [ 887.590330] Key type .llcrypt unregistered [ 887.592147] Key type ._llcrypt unregistered [ 903.769776] Key type ._llcrypt registered [ 903.771756] Key type .llcrypt registered [ 903.846074] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 904.805312] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 904.867230] alg: No test for adler32 (adler32-zlib) [ 906.031837] Lustre: Lustre: Build Version: 2.17.56_50_g8cc021f [ 906.195136] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 907.884090] Key type lgssc registered [ 908.968235] Lustre: Echo OBD driver; http://www.lustre.org/ [ 915.710647] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 915.726651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 917.051645] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 920.901894] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 927.200603] LustreError: 34966:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 927.217468] LustreError: 34966:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 928.819400] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 929.233400] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 933.672212] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 940.700513] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 945.417203] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 950.729537] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 950.981430] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 956.669854] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 962.291872] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 963.343822] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 972.790245] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 972.825762] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 977.892776] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 977.902314] Lustre: Skipped 2 previous similar messages [ 978.487824] Lustre: server umount lustre-OST0000 complete [ 983.009452] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 983.016411] Lustre: Skipped 1 previous similar message [ 983.022058] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 983.026484] Lustre: Skipped 1 previous similar message [ 988.128914] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 988.133673] Lustre: Skipped 1 previous similar message [ 988.503618] Lustre: server umount lustre-MDT0000 complete [ 991.660821] LustreError: 35701:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786339402 with bad export cookie 311899642860329413 [ 991.667658] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 991.705704] LustreError: 35701:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 991.945622] Lustre: server umount lustre-MDT0001 complete [ 1002.492759] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 1004.690972] Key type lgssc unregistered [ 1004.977821] LNet: 38501:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1004.982741] LNetError: 38501:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1006.052942] LNet: Removed LNI 192.168.203.123@tcp [ 1006.724211] Key type .llcrypt unregistered [ 1006.726506] Key type ._llcrypt unregistered [ 1027.895161] Key type ._llcrypt registered [ 1027.896590] Key type .llcrypt registered [ 1027.973573] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 1029.282271] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1029.326954] alg: No test for adler32 (adler32-zlib) [ 1030.434898] Lustre: Lustre: Build Version: 2.17.56_50_g8cc021f [ 1030.663325] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 1032.396822] Key type lgssc registered [ 1033.348735] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1040.635796] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1040.660560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1042.059569] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1046.134183] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1051.619262] LustreError: 39923:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1051.629631] LustreError: 39923:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1053.405958] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1053.723139] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1058.471715] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1066.157570] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1071.424305] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1076.641968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1076.842757] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1082.297790] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1087.124382] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1090.068234] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1097.531117] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.23@tcp (stopping) [ 1097.706804] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1097.738642] Lustre: Skipped 1 previous similar message [ 1102.003922] Lustre: server umount lustre-MDT0000 complete [ 1102.625846] LustreError: 39918:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1105.162759] LustreError: 39904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786339515 with bad export cookie 9879653564869664922 [ 1105.171050] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1105.179747] LustreError: 39904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1107.744965] LustreError: 39917:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1107.775644] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.23@tcp (stopping) [ 1107.782065] LustreError: 39917:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1107.798560] Lustre: Skipped 3 previous similar messages [ 1107.946535] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1107.957616] Lustre: Skipped 1 previous similar message [ 1111.010151] LustreError: 39917:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1111.011672] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1111.023291] LustreError: 39917:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1111.032573] Lustre: Skipped 1 previous similar message [ 1111.610238] Lustre: server umount lustre-MDT0001 complete [ 1133.378372] Lustre: server umount lustre-OST0000 complete [ 1149.201920] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 1151.432979] Key type lgssc unregistered [ 1151.672656] LNet: 43784:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1151.676843] LNetError: 43784:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1151.693238] LNet: Removed LNI 192.168.203.123@tcp [ 1152.537814] Key type .llcrypt unregistered [ 1152.541618] Key type ._llcrypt unregistered [ 1175.123473] Key type ._llcrypt registered [ 1175.125026] Key type .llcrypt registered [ 1175.415454] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 1176.934250] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1176.962669] alg: No test for adler32 (adler32-zlib) [ 1177.930382] Lustre: Lustre: Build Version: 2.17.56_50_g8cc021f [ 1178.196337] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 1179.855168] Key type lgssc registered [ 1180.810699] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1186.817840] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1186.839958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1188.183964] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1192.488686] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1197.537730] LustreError: 45199:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1197.565561] LustreError: 45199:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1201.721712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1202.144946] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1206.274610] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1212.260536] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1217.490082] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1223.957615] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1224.186527] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1230.933821] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1233.429291] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1237.569401] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1243.620826] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1243.632141] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1243.646388] Lustre: Skipped 1 previous similar message [ 1243.674480] Lustre: Skipped 1 previous similar message [ 1245.496191] Lustre: lustre-OST0000: Not available for connect from 192.168.203.23@tcp (stopping) [ 1248.738773] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1248.750701] Lustre: Skipped 1 previous similar message [ 1249.629957] Lustre: server umount lustre-OST0000 complete [ 1266.906591] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1266.913517] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1266.923189] Lustre: Skipped 1 previous similar message [ 1266.947035] Lustre: Skipped 1 previous similar message [ 1272.288567] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1272.292635] Lustre: Skipped 1 previous similar message [ 1273.355956] Lustre: server umount lustre-MDT0000 complete [ 1274.345984] LustreError: 45199:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1274.375483] LustreError: 45199:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1277.200119] LustreError: 45180:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786339687 with bad export cookie 35378315298079465 [ 1277.230323] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1277.238356] LustreError: 45180:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1283.872619] Lustre: server umount lustre-MDT0001 complete [ 1294.831479] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 1297.645207] Key type lgssc unregistered [ 1297.839656] LNet: 48912:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1297.855711] LNetError: 48912:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1297.875795] LNet: Removed LNI 192.168.203.123@tcp [ 1299.004873] Key type .llcrypt unregistered [ 1299.007206] Key type ._llcrypt unregistered [ 1324.954607] Key type ._llcrypt registered [ 1324.957301] Key type .llcrypt registered [ 1325.169370] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 1328.459098] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1328.536987] alg: No test for adler32 (adler32-zlib) [ 1329.852909] Lustre: Lustre: Build Version: 2.17.56_50_g8cc021f [ 1330.150744] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 1331.855239] Key type lgssc registered [ 1332.871543] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1341.091219] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1341.103816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1342.477202] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1346.778603] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1352.676654] LustreError: 50332:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1352.700568] LustreError: 50332:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1353.683101] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1353.896336] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1358.054497] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1364.715263] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1370.624214] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1376.929461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1377.166704] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1383.642564] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1387.449597] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1391.690591] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1398.096742] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1401.328206] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1413.585256] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-10 1h30m22s [ 1415.226953] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1418.223276] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1418.227126] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1418.233615] Lustre: Skipped 2 previous similar messages [ 1418.244813] Lustre: Skipped 2 previous similar messages [ 1419.297501] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1419.316545] Lustre: Skipped 2 previous similar messages [ 1421.172694] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.23@tcp (stopping) [ 1421.182870] Lustre: Skipped 1 previous similar message [ 1423.101122] Lustre: server umount lustre-MDT0000 complete [ 1425.389991] LustreError: 50327:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1425.411317] LustreError: 50327:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1425.993982] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1431.522233] LustreError: 51061:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1431.549484] LustreError: 51061:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1433.234776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1433.322783] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1433.633874] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1433.741891] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1434.715304] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1437.910858] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1441.780995] Lustre: DEBUG MARKER: Wait for df (36755) ... [ 1443.431052] Lustre: DEBUG MARKER: done [ 1448.928476] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1448.936484] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1448.972669] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1449.956676] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1453.164451] Lustre: server umount lustre-OST0000 complete [ 1460.196736] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1460.215726] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1460.220440] Lustre: Skipped 2 previous similar messages [ 1462.979908] Lustre: server umount lustre-MDT0000 complete [ 1465.965059] LustreError: 50313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786339876 with bad export cookie 16283925439291985608 [ 1465.974942] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1465.979798] LustreError: 50313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1466.216813] Lustre: server umount lustre-MDT0001 complete [ 1477.600242] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 1480.162483] Key type lgssc unregistered [ 1480.553358] LNet: 55256:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1480.575705] LNetError: 55256:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1480.608979] LNet: Removed LNI 192.168.203.123@tcp [ 1482.018808] Key type .llcrypt unregistered [ 1482.022823] Key type ._llcrypt unregistered [ 1520.634589] Key type ._llcrypt registered [ 1520.638823] Key type .llcrypt registered [ 1520.737727] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 1521.979473] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1522.410719] alg: No test for adler32 (adler32-zlib) [ 1524.021500] Lustre: Lustre: Build Version: 2.17.56_50_g8cc021f [ 1524.423240] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 1526.055206] Key type lgssc registered [ 1526.986901] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1534.178651] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1534.201801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1535.415571] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1535.433599] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1535.576271] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1539.316264] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1547.434155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1547.504471] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1547.533599] Lustre: 57466:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1547.876885] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1552.247440] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1558.990691] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1564.699641] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1570.728840] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1570.855058] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1570.972625] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1576.412900] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1579.286101] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1584.074131] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1590.563852] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-10 1h33m20s [ 1592.020114] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1599.912865] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1601.934248] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1601.958992] LustreError: 57472:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1622.398814] Lustre: lustre-MDT0000: Client 15a4df73-fee0-48b2-9a7d-56bd90383d90 (at 192.168.203.23@tcp) reconnecting [ 1642.015119] LustreError: 57472:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1643.780259] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1645.963099] Lustre: DEBUG MARKER: done [ 1651.168121] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1651.179030] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1651.196326] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1651.210641] Lustre: Skipped 1 previous similar message [ 1656.292805] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1656.301120] Lustre: Skipped 2 previous similar messages [ 1657.264460] Lustre: server umount lustre-OST0000 complete [ 1660.397141] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1660.413814] LustreError: Skipped 1 previous similar message [ 1660.430709] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1660.461792] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1666.598232] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1666.605596] Lustre: Skipped 4 previous similar messages [ 1666.731543] Lustre: server umount lustre-MDT0000 complete [ 1670.789589] LustreError: 57457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786340081 with bad export cookie 17593853312281656005 [ 1670.810452] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1670.819434] LustreError: 57457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1671.198413] Lustre: server umount lustre-MDT0001 complete [ 1681.738332] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 1683.813781] Key type lgssc unregistered [ 1684.093646] LNet: 61798:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1684.099310] LNetError: 61798:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1684.112812] LNet: Removed LNI 192.168.203.123@tcp [ 1685.025533] Key type .llcrypt unregistered [ 1685.028242] Key type ._llcrypt unregistered [ 1725.493992] Key type ._llcrypt registered [ 1725.496275] Key type .llcrypt registered [ 1725.652803] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 1726.897528] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1727.076429] alg: No test for adler32 (adler32-zlib) [ 1728.483728] Lustre: Lustre: Build Version: 2.17.56_50_g8cc021f [ 1728.741221] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 1730.431427] Key type lgssc registered [ 1731.371653] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1734.042412] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1734.056308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1735.198235] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1735.216362] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1735.327947] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1736.192807] Lustre: Failing over lustre-MDT0000 [ 1736.367429] Lustre: server umount lustre-MDT0000 complete [ 1742.429786] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 01:35:52 (1786340152) [ 1748.172017] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1748.468102] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1751.943576] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1758.649502] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1758.716154] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1758.735256] Lustre: 64598:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1758.933739] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1762.780668] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1768.919155] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1773.406869] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1778.255197] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1778.466958] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1778.621790] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1783.740570] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1785.483380] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1790.524559] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1795.959472] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1801.187297] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1801.190038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1801.193623] Lustre: Skipped 1 previous similar message [ 1801.208531] Lustre: Skipped 2 previous similar messages [ 1803.329947] Lustre: server umount lustre-MDT0000 complete [ 1806.407318] LustreError: 64591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786340216 with bad export cookie 9145367745362983978 [ 1806.418828] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1806.630739] Lustre: server umount lustre-MDT0001 complete [ 1809.533931] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1813.673985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1820.988342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1827.807150] Lustre: 63520:0:(client.c:2506:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786340222/real 1786340222] req@00000000ddb074ad x1873113397628800/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786340238 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1827.824276] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1827.833262] Lustre: Skipped 1 previous similar message [ 1831.968489] LustreError: 63519:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000cff7a363 x1873113397629568/t0(0) o250->MGC192.168.203.123@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1832.282293] LustreError: 66446:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1832.307521] LustreError: 66446:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 1832.359244] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1832.462629] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1833.377092] Lustre: 63520:0:(client.c:2506:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786340227/real 1786340227] req@00000000b4b43f60 x1873113397628928/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786340243 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1836.369779] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1836.386299] LustreError: 66447:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1837.479204] Lustre: 63522:0:(client.c:2506:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786340232/real 1786340232] req@000000006cc746bf x1873113397629184/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786340248 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1838.441218] LustreError: 68364:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1838.464380] LustreError: 68364:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1842.655117] Lustre: 63523:0:(client.c:2506:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786340237/real 1786340237] req@0000000085f7b2e6 x1873113397629312/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786340253 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1843.673401] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1843.698176] LustreError: 68364:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1843.713863] LustreError: 68364:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1843.928438] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1848.644557] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1849.337978] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1855.444133] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1856.557304] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1860.229262] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1866.726293] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1866.739411] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1866.745556] Lustre: Skipped 2 previous similar messages [ 1866.756467] Lustre: Skipped 1 previous similar message [ 1867.397285] Lustre: server umount lustre-MDT0000 complete [ 1870.356949] LustreError: 64592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786340280 with bad export cookie 9145367745362986253 [ 1870.363647] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1870.366705] LustreError: 64592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1870.590857] Lustre: server umount lustre-MDT0001 complete [ 1875.659954] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1882.631640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1888.223130] Lustre: 63521:0:(client.c:2506:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786340282/real 1786340282] req@000000006b2b82dd x1873113397654016/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786340298 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1888.253832] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1894.431785] LustreError: 63519:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000007547902a x1873113397655168/t0(0) o250->MGC192.168.203.123@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1894.636105] LustreError: 66447:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1894.698070] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1894.739744] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1898.623441] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1900.899262] Lustre: 63521:0:(client.c:2506:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786340295/real 1786340295] req@00000000e0600e3f x1873113397654656/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786340311 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1900.915025] Lustre: 63521:0:(client.c:2506:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 1903.079924] LustreError: 66447:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1903.099641] LustreError: 66447:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1905.335757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1906.661947] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1909.539387] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1916.072799] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1921.687819] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1927.215358] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1927.218867] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1927.225190] Lustre: Skipped 2 previous similar messages [ 1927.233339] Lustre: Skipped 1 previous similar message [ 1929.060328] Lustre: server umount lustre-MDT0000 complete [ 1932.261301] LustreError: 71045:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1932.329195] LustreError: 71045:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1932.361690] LustreError: 64592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786340342 with bad export cookie 9145367745362987632 [ 1932.372419] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1932.455783] LustreError: 64592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1932.832531] Lustre: server umount lustre-MDT0001 complete [ 1937.967749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1941.693803] Lustre: DEBUG MARKER: files compared the same [ 1953.400798] Lustre: server umount lustre-OST0000 complete [ 1966.189852] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 1968.459925] Key type lgssc unregistered [ 1968.707736] LNet: 74321:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1968.712549] LNetError: 74321:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1968.724564] LNet: Removed LNI 192.168.203.123@tcp [ 1969.457835] Key type .llcrypt unregistered [ 1969.461393] Key type ._llcrypt unregistered [ 1988.666210] Key type ._llcrypt registered [ 1988.669669] Key type .llcrypt registered [ 1988.762761] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 1990.039577] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1990.085116] alg: No test for adler32 (adler32-zlib) [ 1991.065082] Lustre: Lustre: Build Version: 2.17.56_50_g8cc021f [ 1991.264453] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 1992.991348] Key type lgssc registered [ 1994.005971] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2000.127978] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2000.144725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2001.560616] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2005.101401] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug +malloc all [ 2012.117701] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2012.128689] LustreError: 75718:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2012.146348] LustreError: 75718:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2012.321711] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2016.749509] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug +malloc all [ 2022.320203] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2027.415276] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2032.905828] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2033.139908] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2038.110302] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug +malloc all [ 2044.010381] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2046.442703] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2056.674080] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2056.675212] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2056.694728] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2056.728872] Lustre: Skipped 1 previous similar message [ 2061.483420] Lustre: server umount lustre-OST0000 complete [ 2066.912046] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2066.923271] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2066.934066] Lustre: Skipped 2 previous similar messages [ 2066.945381] Lustre: Skipped 1 previous similar message [ 2070.844246] Lustre: server umount lustre-MDT0000 complete [ 2072.032899] LustreError: 75714:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2072.045549] LustreError: 75714:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2073.964343] LustreError: 75699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786340484 with bad export cookie 18117403837063779735 [ 2073.969693] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2073.973531] LustreError: 75699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2074.237964] Lustre: server umount lustre-MDT0001 complete [ 2083.307845] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 2085.482954] Key type lgssc unregistered [ 2085.767486] LNet: 79252:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2085.782667] LNetError: 79252:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2085.796679] LNet: Removed LNI 192.168.203.123@tcp [ 2086.550654] Key type .llcrypt unregistered [ 2086.553134] Key type ._llcrypt unregistered [ 2109.395199] Key type ._llcrypt registered [ 2109.397166] Key type .llcrypt registered [ 2109.510691] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 2110.637109] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2110.681132] alg: No test for adler32 (adler32-zlib) [ 2111.752884] Lustre: Lustre: Build Version: 2.17.56_50_g8cc021f [ 2111.927318] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 2113.639172] Key type lgssc registered [ 2114.579529] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2121.071265] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2121.088210] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2136.608562] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2141.727386] LustreError: 80641:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.123@tcp: failed processing log, type 4: rc = -110 [ 2168.351207] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2173.745891] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2180.292391] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2186.674895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2186.745323] Lustre: *** cfs_fail_loc=706, val=0*** [ 2187.052759] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2187.100058] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2190.542829] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2192.355618] LustreError: 80666:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2192.366445] LustreError: 80666:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2197.477600] LustreError: 80667:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2197.505878] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2197.777930] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2201.553121] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2208.066510] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2212.539527] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2217.989238] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2217.994719] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2218.002761] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2220.466347] Lustre: server umount lustre-OST0000 complete [ 2228.706572] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2228.706997] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2228.723514] Lustre: Skipped 1 previous similar message [ 2228.724687] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2228.729793] LustreError: Skipped 1 previous similar message [ 2228.740113] Lustre: Skipped 3 previous similar messages [ 2230.248625] Lustre: server umount lustre-MDT0000 complete [ 2233.022639] LustreError: 80647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786340643 with bad export cookie 15956918328577258328 [ 2233.029244] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2233.029959] LustreError: 80647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2233.237216] Lustre: server umount lustre-MDT0001 complete [ 2241.929623] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 2243.907525] Key type lgssc unregistered [ 2244.187847] LNet: 84255:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2244.200908] LNetError: 84255:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2244.210692] LNet: Removed LNI 192.168.203.123@tcp [ 2244.886322] Key type .llcrypt unregistered [ 2244.887978] Key type ._llcrypt unregistered [ 2265.612750] Key type ._llcrypt registered [ 2265.615388] Key type .llcrypt registered [ 2265.689687] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 2266.710348] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2266.726045] alg: No test for adler32 (adler32-zlib) [ 2267.865407] Lustre: Lustre: Build Version: 2.17.56_50_g8cc021f [ 2268.073668] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 2269.711184] Key type lgssc registered [ 2270.724315] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2277.467574] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2277.554635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2283.065436] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2289.110952] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2289.599243] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2293.390382] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2294.755376] LustreError: 86338:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2294.766861] LustreError: 86338:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2299.614774] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2299.839273] LustreError: 86337:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2299.879168] LustreError: 86337:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2299.895870] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2300.897788] LustreError: 86336:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2300.916898] LustreError: 86336:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2304.684612] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2306.021252] LustreError: 86337:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2306.063018] LustreError: 86337:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2310.720436] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2310.733141] Lustre: MGS: Client 36f802db-621b-49ce-a157-68b1e959cde9 (at 0@lo) reconnecting [ 2310.752174] LustreError: 86352:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2310.899916] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2310.943622] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2314.283908] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2326.497946] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2326.503437] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2326.511642] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2326.517765] Lustre: Skipped 1 previous similar message [ 2326.526941] Lustre: Skipped 1 previous similar message [ 2330.750845] Lustre: server umount lustre-OST0000 complete [ 2336.740745] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2336.743187] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2336.767855] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2336.781916] Lustre: Skipped 1 previous similar message [ 2339.797762] Lustre: server umount lustre-MDT0000 complete [ 2341.858565] LustreError: 87882:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2341.872726] LustreError: 87882:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2343.088844] Lustre: server umount lustre-MDT0001 complete [ 2345.789950] Lustre: server umount MGS complete [ 2353.972701] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 01:46:04 (1786340764) [ 2375.815542] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 2384.335113] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 2420.645400] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2425.725863] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2433.773172] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2433.994833] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2434.068099] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2434.122963] Lustre: lustre-MDT0001: new disk, initializing [ 2434.201993] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2434.224851] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2434.232736] Lustre: 93506:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2434.245961] LustreError: 93506:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2438.585601] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2447.685824] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2447.963926] Lustre: lustre-OST0000: new disk, initializing [ 2447.967104] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2447.979751] Lustre: 94271:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2448.027684] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2453.708809] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2456.096531] Lustre: 94710:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000aa0061b5 x1873113962812800/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1786340882 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2461.958628] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2468.249588] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2468.261050] Lustre: MGS: Client b4a38dc5-6dba-4db1-85c7-87e2a397a68d (at 0@lo) reconnecting [ 2468.316065] Lustre: 95316:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2468.345196] LustreError: 93492:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2468.379617] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2468.442399] Lustre: lustre-MDT0000: new disk, initializing [ 2468.582887] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2468.651846] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2468.786631] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2469.728489] Lustre: 95341:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000a95efd80 x1873113962819200/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1786340896 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2473.130727] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2476.511670] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2478.889312] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2478.902158] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2479.075834] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2503.649173] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2503.658385] Lustre: Skipped 2 previous similar messages [ 2503.664400] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2503.668305] Lustre: Skipped 1 previous similar message [ 2507.833763] Lustre: server umount lustre-OST0000 complete [ 2513.917926] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2513.923156] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2513.933411] Lustre: Skipped 1 previous similar message [ 2513.982259] Lustre: Skipped 1 previous similar message [ 2517.489356] Lustre: server umount lustre-MDT0000 complete [ 2519.018487] LustreError: 93478:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2527.963513] Lustre: server umount lustre-MDT0001 complete [ 2532.392405] Lustre: server umount MGS complete [ 2543.506410] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 01:49:13 (1786340953) [ 2558.901370] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 2561.358194] Key type lgssc unregistered [ 2561.651200] LNet: 97843:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2561.656338] LNetError: 97843:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2561.667779] LNet: Removed LNI 192.168.203.123@tcp [ 2562.200573] Key type .llcrypt unregistered [ 2562.202143] Key type ._llcrypt unregistered [ 2577.066460] Key type ._llcrypt registered [ 2577.070692] Key type .llcrypt registered [ 2577.199247] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 2577.966911] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2577.995466] alg: No test for adler32 (adler32-zlib) [ 2579.168675] Lustre: Lustre: Build Version: 2.17.56_50_g8cc021f [ 2579.477623] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 2581.271263] Key type lgssc registered [ 2582.342459] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2588.391527] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2588.407292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2589.646648] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2593.580396] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2599.910268] LustreError: 99046:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2599.920630] LustreError: 99046:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2601.122317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2601.405708] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2605.729830] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2612.968677] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2618.350625] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2626.234259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2626.288836] LustreError: 100869:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2626.596237] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2632.544565] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2634.742744] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 2638.989422] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2647.012637] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2647.021420] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2647.038241] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2650.089455] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2650.089749] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2650.089888] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2652.162474] Lustre: server umount lustre-OST0000 complete [ 2657.760446] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2657.766547] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2657.777528] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2657.783386] Lustre: Skipped 1 previous similar message [ 2660.321479] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2660.323354] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2664.389937] Lustre: server umount lustre-MDT0000 complete [ 2665.443213] LustreError: 99045:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2667.990291] LustreError: 99027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786341078 with bad export cookie 5428676949619122642 [ 2667.999148] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2667.999974] LustreError: 99027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2668.264416] Lustre: server umount lustre-MDT0001 complete [ 2677.622861] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 2680.621712] Key type lgssc unregistered [ 2680.864707] LNet: 102583:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2680.870063] LNetError: 102583:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2680.880194] LNet: Removed LNI 192.168.203.123@tcp [ 2681.521927] Key type .llcrypt unregistered [ 2681.523996] Key type ._llcrypt unregistered [ 2699.013489] Key type ._llcrypt registered [ 2699.016049] Key type .llcrypt registered [ 2699.158942] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 2701.424334] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2701.454812] alg: No test for adler32 (adler32-zlib) [ 2703.059700] Lustre: Lustre: Build Version: 2.17.56_50_g8cc021f [ 2703.594675] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 2705.215620] Key type lgssc registered [ 2706.009762] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2712.267000] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2712.275770] LustreError: 103665:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 2712.303071] LustreError: 103667:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 2712.311112] LustreError: 103665:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4984 [ 2712.323937] LustreError: 103665:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2712.337687] LustreError: 103665:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2712.352697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2713.733813] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2720.762728] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2721.155863] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2725.836288] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2729.953951] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing lsmod [ 2732.390162] LustreError: 104853:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 2732.579862] LustreError: 104873:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 2732.586681] LustreError: 104853:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4807 [ 2732.604205] LustreError: 104853:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2732.616217] LustreError: 104853:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2732.732220] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2733.228747] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2736.609473] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2736.618974] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2736.635552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2738.663155] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2738.669193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2738.686197] Lustre: Skipped 1 previous similar message [ 2738.690234] Lustre: Skipped 1 previous similar message [ 2742.417286] Lustre: server umount lustre-MDT0000 complete [ 2743.893759] LustreError: 103705:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2743.929921] LustreError: 103705:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2745.899601] LustreError: 103680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786341156 with bad export cookie 9430288975592367368 [ 2745.908187] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2745.909860] LustreError: 103680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2745.921674] Lustre: 103697:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.123@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 2746.109676] Lustre: server umount lustre-MDT0001 complete [ 2759.570425] Lustre: server umount lustre-OST0000 complete [ 2766.885579] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2767.385617] LustreError: 105937:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2767.479046] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2771.801069] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2772.963208] LustreError: 105938:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2778.084515] LustreError: 105937:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2779.047254] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2779.384194] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2784.188634] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2791.154948] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2796.107912] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2801.922141] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2802.134614] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2807.249917] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2813.013852] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2813.423893] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 2819.553363] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2819.564820] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2819.589230] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2819.599321] Lustre: Skipped 1 previous similar message [ 2823.651726] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2823.654710] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2825.544299] Lustre: server umount lustre-OST0000 complete [ 2830.816510] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2830.824693] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2830.843320] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2830.848388] Lustre: Skipped 1 previous similar message [ 2835.702725] Lustre: server umount lustre-MDT0000 complete [ 2839.014641] LustreError: 105937:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2839.029873] LustreError: 105937:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2839.324083] LustreError: 105919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786341249 with bad export cookie 9430288975592368054 [ 2839.331209] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2839.342621] LustreError: 105919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2839.731980] Lustre: server umount lustre-MDT0001 complete [ 2849.592796] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 2853.092287] Key type lgssc unregistered [ 2853.728813] LNet: 109475:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2853.744543] LNetError: 109475:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2853.761774] LNet: Removed LNI 192.168.203.123@tcp [ 2854.303699] Key type .llcrypt unregistered [ 2854.305484] Key type ._llcrypt unregistered [ 2877.811345] Key type ._llcrypt registered [ 2877.813564] Key type .llcrypt registered [ 2877.927903] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 2879.212438] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2879.465415] alg: No test for adler32 (adler32-zlib) [ 2880.541501] Lustre: Lustre: Build Version: 2.17.56_50_g8cc021f [ 2880.706112] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 2882.337636] Key type lgssc registered [ 2883.283933] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2889.509837] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2889.528425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2890.867499] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2894.428306] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2900.960155] LustreError: 110895:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2900.983642] LustreError: 110895:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2901.546302] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2901.807239] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2905.700150] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2911.282676] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2916.139767] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2922.193954] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2922.416615] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2928.256460] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2935.383797] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2935.726137] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 2940.633256] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 2943.605255] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 2947.040072] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2947.044759] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2947.059462] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2951.139431] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2951.146360] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2951.155046] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2951.177899] Lustre: Skipped 1 previous similar message [ 2952.753336] Lustre: server umount lustre-OST0000 complete [ 2956.256960] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2956.259967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2956.271779] Lustre: Skipped 1 previous similar message [ 2956.285019] Lustre: Skipped 1 previous similar message [ 2961.376809] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2961.390862] Lustre: Skipped 1 previous similar message [ 2962.319704] Lustre: server umount lustre-MDT0000 complete [ 2965.123925] LustreError: 110874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786341375 with bad export cookie 52204556018656118 [ 2965.145737] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2965.147725] LustreError: 110874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2965.398778] Lustre: server umount lustre-MDT0001 complete [ 2974.923636] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 2976.911672] Key type lgssc unregistered [ 2977.153153] LNet: 114532:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2977.160623] LNetError: 114532:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2977.179353] LNet: Removed LNI 192.168.203.123@tcp [ 2977.755653] Key type .llcrypt unregistered [ 2977.758368] Key type ._llcrypt unregistered [ 2991.826589] Key type ._llcrypt registered [ 2991.828696] Key type .llcrypt registered [ 2991.912865] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 2992.755060] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2992.846558] alg: No test for adler32 (adler32-zlib) [ 2993.833921] Lustre: Lustre: Build Version: 2.17.56_50_g8cc021f [ 2994.007707] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 2995.631195] Key type lgssc registered [ 2996.498561] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3002.377537] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3002.411624] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3003.772530] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3007.403249] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3013.964924] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3014.116369] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3014.282269] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3017.636585] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3023.289292] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3029.061437] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3034.375298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3034.602458] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3039.252972] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3044.446672] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3048.951657] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3059.168867] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3059.173939] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3059.176243] LustreError: Skipped 1 previous similar message [ 3059.178096] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3059.186543] Lustre: Skipped 1 previous similar message [ 3061.097198] Lustre: server umount lustre-OST0000 complete [ 3064.294137] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3064.297955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3064.303570] Lustre: Skipped 1 previous similar message [ 3064.317895] Lustre: Skipped 2 previous similar messages [ 3069.409053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3069.417027] Lustre: Skipped 1 previous similar message [ 3070.220747] Lustre: server umount lustre-MDT0000 complete [ 3072.919861] LustreError: 115715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786341483 with bad export cookie 13169376726777809381 [ 3072.922669] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3072.925295] LustreError: 115715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3073.156360] Lustre: server umount lustre-MDT0001 complete [ 3080.792661] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 3082.961662] Key type lgssc unregistered [ 3083.176256] LNet: 119266:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3083.188963] LNetError: 119266:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3083.205551] LNet: Removed LNI 192.168.203.123@tcp [ 3083.871869] Key type .llcrypt unregistered [ 3083.875350] Key type ._llcrypt unregistered [ 3102.099915] Key type ._llcrypt registered [ 3102.106269] Key type .llcrypt registered [ 3102.211371] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 3103.035200] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3103.086061] alg: No test for adler32 (adler32-zlib) [ 3104.122876] Lustre: Lustre: Build Version: 2.17.56_50_g8cc021f [ 3104.305653] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 3105.935767] Key type lgssc registered [ 3106.963448] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3112.358826] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3112.368823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3113.661968] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3116.694380] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3122.142114] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3122.344343] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3125.380521] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3130.452112] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3134.387286] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3139.123962] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3139.315486] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3143.120179] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3147.341122] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3158.763487] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3162.271433] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3164.296738] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3164.301811] Lustre: Skipped 2 previous similar messages [ 3166.526950] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3166.531120] Lustre: Skipped 2 previous similar messages [ 3170.324588] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3170.349298] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3173.748926] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.23@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3173.754638] Lustre: Skipped 2 previous similar messages [ 3180.037335] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.23@tcp 192.168.0.[2,10]@tcp [ 3183.192884] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.23/24@tcp in log lustre-MDT0000 [ 3183.201114] Lustre: Skipped 2 previous similar messages [ 3189.837077] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.23/24@tcp [ 3189.848258] Lustre: Skipped 1 previous similar message [ 3194.850226] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3194.853072] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3194.854200] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3194.860358] Lustre: Skipped 1 previous similar message [ 3199.735398] Lustre: server umount lustre-OST0000 complete [ 3204.576462] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3204.580301] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3204.586793] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3204.590856] Lustre: Skipped 1 previous similar message [ 3210.209458] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3210.212245] Lustre: Skipped 3 previous similar messages [ 3215.328853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3215.334774] Lustre: Skipped 1 previous similar message [ 3216.351141] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3216.452324] Lustre: server umount lustre-MDT0000 complete [ 3218.609804] LustreError: 120642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786341629 with bad export cookie 8893096837265462593 [ 3218.612850] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3218.619592] LustreError: 120642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3218.785159] Lustre: server umount lustre-MDT0001 complete [ 3225.934440] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 3227.799234] Key type lgssc unregistered [ 3228.021828] LNet: 125858:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3228.026853] LNetError: 125858:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3228.039195] LNet: Removed LNI 192.168.203.123@tcp [ 3228.658328] Key type .llcrypt unregistered [ 3228.663743] Key type ._llcrypt unregistered [ 3247.477509] Key type ._llcrypt registered [ 3247.479600] Key type .llcrypt registered [ 3247.566280] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 3248.191860] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3248.233731] alg: No test for adler32 (adler32-zlib) [ 3249.231611] Lustre: Lustre: Build Version: 2.17.56_50_g8cc021f [ 3249.413168] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 3251.047293] Key type lgssc registered [ 3251.768627] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3256.544744] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3256.557967] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3257.763584] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3257.770057] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3257.778349] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.23/24@tcp [ 3260.345715] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3265.618849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3265.807611] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3265.844379] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.23/24@tcp [ 3265.852658] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3268.455608] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3272.649879] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3276.157215] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3280.254902] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3280.451701] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3282.552815] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 3284.424632] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3288.461952] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3298.273385] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3298.278476] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3298.286427] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3301.344259] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3301.348891] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3301.349580] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3301.361389] Lustre: Skipped 1 previous similar message [ 3303.782315] Lustre: server umount lustre-OST0000 complete [ 3306.465606] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3306.478135] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3306.482828] Lustre: Skipped 1 previous similar message [ 3311.584916] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3311.590352] Lustre: Skipped 1 previous similar message [ 3316.709607] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3316.713534] Lustre: Skipped 1 previous similar message [ 3320.287156] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3320.374965] Lustre: server umount lustre-MDT0000 complete [ 3321.827227] LustreError: 127412:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3321.838600] LustreError: 127412:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3322.228750] LustreError: 127393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786341732 with bad export cookie 15340960094933297058 [ 3322.232417] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3322.237310] LustreError: 127393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3322.401688] Lustre: server umount lustre-MDT0001 complete [ 3327.514756] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 3328.985315] Key type lgssc unregistered [ 3329.159638] LNet: 130998:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3329.168816] LNetError: 130998:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3329.179529] LNet: Removed LNI 192.168.203.123@tcp [ 3329.618519] Key type .llcrypt unregistered [ 3329.620704] Key type ._llcrypt unregistered [ 3342.968424] Key type ._llcrypt registered [ 3342.971168] Key type .llcrypt registered [ 3343.033660] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 3351.641526] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 3352.258932] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3352.281505] alg: No test for adler32 (adler32-zlib) [ 3353.209308] Lustre: Lustre: Build Version: 2.17.56_50_g8cc021f [ 3353.356789] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 3354.959186] Key type lgssc registered [ 3355.722870] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3375.394351] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3375.411297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3376.579814] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3376.600355] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3376.659372] Lustre: lustre-MDT0000: new disk, initializing [ 3376.721044] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3376.732040] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3379.164450] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3385.400271] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3385.502868] Lustre: 134373:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3385.562692] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3385.571292] Lustre: Skipped 1 previous similar message [ 3385.624403] Lustre: lustre-MDT0001: new disk, initializing [ 3385.706564] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3385.722779] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3385.727863] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3388.170987] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3394.123812] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3397.670606] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3401.654599] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3401.875385] Lustre: lustre-OST0000: new disk, initializing [ 3401.877967] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3401.880802] Lustre: 136341:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3401.918198] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3402.279510] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3402.287275] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3402.322686] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3405.351368] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3410.958115] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3413.760949] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3413.896649] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3416.568954] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3416.686699] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3422.689494] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3422.689696] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3422.695102] LustreError: Skipped 1 previous similar message [ 3422.701386] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3422.706875] Lustre: Skipped 1 previous similar message [ 3423.855804] Lustre: server umount lustre-OST0000 complete [ 3426.784220] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3426.787230] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3426.791717] Lustre: Skipped 1 previous similar message [ 3426.794446] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3427.808509] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3427.811241] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3432.114217] Lustre: server umount lustre-MDT0000 complete [ 3432.928533] LustreError: 134380:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3432.944036] LustreError: 134380:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3434.313500] LustreError: 134365:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786341844 with bad export cookie 15344590151848129864 [ 3434.320353] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3434.323530] LustreError: 134365:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3434.503778] Lustre: server umount lustre-MDT0001 complete [ 3439.143788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3439.385486] LustreError: 138559:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3439.425880] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3441.839511] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3444.706793] LustreError: 138560:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3446.666630] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3446.884334] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3449.512192] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3453.665538] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3456.996260] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3460.719445] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3460.892995] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3464.085888] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3468.105303] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3470.644738] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3470.779338] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3473.011464] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3473.152325] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3478.743164] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3478.845298] Lustre: lustre-OST0001: new disk, initializing [ 3478.847468] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3478.852673] Lustre: 141925:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3478.918639] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3482.175248] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3487.260297] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3487.278555] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3487.289337] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3487.335937] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3488.425639] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 3490.829961] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3490.952527] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3493.668956] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff994458ed5800.ost_server_uuid 50 [ 3494.366566] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff994458ed5800.ost_server_uuid in FULL state after 0 sec [ 3497.440971] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3497.441490] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3497.444707] LustreError: Skipped 1 previous similar message [ 3497.452176] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3497.457980] Lustre: Skipped 2 previous similar messages [ 3502.562136] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3502.565073] Lustre: Skipped 1 previous similar message [ 3503.318731] Lustre: server umount lustre-OST0001 complete [ 3507.683843] LustreError: 140378:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3507.686365] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3507.700540] LustreError: 140378:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3507.712385] Lustre: Skipped 2 previous similar messages [ 3511.480610] Lustre: server umount lustre-OST0000 complete [ 3513.312110] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3513.319745] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3513.321837] Lustre: Skipped 2 previous similar messages [ 3517.922941] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3517.930750] Lustre: Skipped 1 previous similar message [ 3519.410614] Lustre: server umount lustre-MDT0000 complete [ 3521.095555] LustreError: 138541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786341931 with bad export cookie 15344590151848130746 [ 3521.097690] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3521.102412] LustreError: 138541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3521.251741] Lustre: server umount lustre-MDT0001 complete [ 3530.291140] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 3531.830805] Key type lgssc unregistered [ 3532.018035] LNet: 144937:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3532.025773] LNetError: 144937:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3532.039921] LNet: Removed LNI 192.168.203.123@tcp [ 3532.497230] Key type .llcrypt unregistered [ 3532.499517] Key type ._llcrypt unregistered [ 3552.063751] Key type ._llcrypt registered [ 3552.065879] Key type .llcrypt registered [ 3552.122200] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 3560.871731] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 3561.530750] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3561.540310] alg: No test for adler32 (adler32-zlib) [ 3562.429912] Lustre: Lustre: Build Version: 2.17.56_50_g8cc021f [ 3562.558931] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 3564.159155] Key type lgssc registered [ 3564.659721] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3581.312750] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3581.320914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3582.438636] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3582.455306] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3582.499241] Lustre: lustre-MDT0000: new disk, initializing [ 3582.540827] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3582.553455] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3584.705205] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3590.709964] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3590.779620] Lustre: 149063:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3590.811269] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3590.813814] Lustre: Skipped 1 previous similar message [ 3590.860669] Lustre: lustre-MDT0001: new disk, initializing [ 3590.928633] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3590.957541] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3590.964185] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3593.204297] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3598.384763] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3600.994085] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3604.084796] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3604.262206] Lustre: lustre-OST0000: new disk, initializing [ 3604.264273] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3604.268408] Lustre: 151030:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3604.308776] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3607.160243] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3612.061121] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3614.220544] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3614.233729] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3614.296801] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3619.621732] Lustre: Failing over lustre-OST0000 [ 3619.697897] Lustre: server umount lustre-OST0000 complete [ 3633.943110] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3634.137354] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3634.143070] Lustre: Skipped 1 previous similar message [ 3634.155466] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3634.166239] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3636.953804] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3638.886677] Lustre: Failing over lustre-MDT0000 [ 3639.023063] Lustre: server umount lustre-MDT0000 complete [ 3639.076872] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3639.264801] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3639.270963] Lustre: Skipped 2 previous similar messages [ 3640.287402] Lustre: 147291:0:(client.c:2506:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786342034/real 1786342034] req@0000000069d9e435 x1873115320710144/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786342050 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3644.180950] LustreError: 149069:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3644.190929] LustreError: 149069:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 3646.431438] Lustre: 147293:0:(client.c:2506:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786342040/real 1786342040] req@0000000044c07266 x1873115320711936/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786342056 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3649.301350] LustreError: 150025:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3649.317344] LustreError: 150025:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3653.755812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3653.823683] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3653.830762] Lustre: 149067:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.123@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 3653.942428] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 3653.969412] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3653.998218] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3654.420367] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3656.133900] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3659.235982] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3659.244834] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 3659.290421] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 3664.352983] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3664.353934] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3664.358023] LustreError: Skipped 1 previous similar message [ 3664.358583] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3664.364187] Lustre: Skipped 1 previous similar message [ 3664.371898] Lustre: Skipped 1 previous similar message [ 3667.037193] Lustre: server umount lustre-OST0000 complete [ 3669.472662] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3669.473199] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3669.474350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3669.485972] Lustre: Skipped 1 previous similar message [ 3674.592064] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3674.594624] Lustre: Skipped 2 previous similar messages [ 3679.714208] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3683.295424] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3683.404529] Lustre: server umount lustre-MDT0000 complete [ 3684.832287] LustreError: 149074:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3684.839515] LustreError: 149074:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 3685.274308] LustreError: 149055:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786342095 with bad export cookie 3651301659286959563 [ 3685.277113] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3685.280795] LustreError: 149055:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3685.426353] Lustre: server umount lustre-MDT0001 complete [ 3690.565514] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 3691.885934] Key type lgssc unregistered [ 3692.038957] LNet: 154679:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3692.048605] LNetError: 154679:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3692.056375] LNet: Removed LNI 192.168.203.123@tcp [ 3692.429517] Key type .llcrypt unregistered [ 3692.431389] Key type ._llcrypt unregistered [ 3715.024581] Key type ._llcrypt registered [ 3715.025925] Key type .llcrypt registered [ 3715.070527] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 3715.512577] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3715.524341] alg: No test for adler32 (adler32-zlib) [ 3716.389616] Lustre: Lustre: Build Version: 2.17.56_50_g8cc021f [ 3716.479552] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 3718.071158] Key type lgssc registered [ 3718.507228] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3721.760302] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3721.771131] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3722.967428] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3724.757566] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3729.619773] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3731.973230] LustreError: 156225:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3731.981310] LustreError: 156225:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3731.986611] LustreError: 156225:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 0, failed: rc = -5 [ 3732.128700] Lustre: server umount lustre-MDT0000 complete [ 3738.129947] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 3742.091716] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 3758.405515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3758.530205] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3758.553817] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3758.604715] Lustre: lustre-MDT0000: new disk, initializing [ 3758.663488] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3758.673458] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3760.845987] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3766.591072] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3766.666063] Lustre: 159522:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3766.704624] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3766.708713] Lustre: Skipped 1 previous similar message [ 3766.749884] Lustre: lustre-MDT0001: new disk, initializing [ 3766.803284] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3766.826399] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3766.831252] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3768.852926] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3773.686492] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3776.451582] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3779.837485] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3779.979667] Lustre: lustre-OST0000: new disk, initializing [ 3779.983529] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3779.992241] Lustre: 161486:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3780.031944] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3781.586267] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3781.599478] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3781.646682] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3782.735315] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3787.786581] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3790.031877] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 3790.928438] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 02:10:01 (1786342201) [ 3889.119839] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3889.120326] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3889.125535] Lustre: Skipped 2 previous similar messages [ 3889.127974] Lustre: Skipped 1 previous similar message [ 3894.240266] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3894.246304] Lustre: Skipped 3 previous similar messages [ 3894.961631] Lustre: server umount lustre-MDT0000 complete [ 3896.710033] LustreError: 159513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786342307 with bad export cookie 3069753323124679553 [ 3896.710501] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3896.716293] LustreError: 159513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3899.360860] LustreError: 159527:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3899.361171] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3899.371427] LustreError: 159527:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3899.383311] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3901.407865] LustreError: 159529:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3901.407892] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3906.528461] LustreError: 160758:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3906.529284] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3911.135248] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3911.249519] Lustre: server umount lustre-MDT0001 complete [ 3919.335959] Lustre: server umount lustre-OST0000 complete [ 3923.791847] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 3928.353667] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 3945.375720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3945.497148] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3945.516964] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3945.564472] Lustre: lustre-MDT0000: new disk, initializing [ 3945.599170] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3945.612668] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3947.467903] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3952.641561] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3952.699665] Lustre: 166121:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3952.725312] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3952.727546] Lustre: Skipped 1 previous similar message [ 3952.768497] Lustre: lustre-MDT0001: new disk, initializing [ 3952.825744] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3952.830341] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3954.660713] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3959.100676] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3961.272248] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3964.122511] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3964.242636] Lustre: lustre-OST0000: new disk, initializing [ 3964.244908] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3964.248920] Lustre: 168090:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3964.299419] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3964.304272] Lustre: Skipped 1 previous similar message [ 3966.802682] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3970.565318] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3970.569126] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3970.629086] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3971.709708] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3973.760841] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3973.880586] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3975.990964] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3976.086575] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3980.769470] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3980.770069] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3980.773239] LustreError: Skipped 1 previous similar message [ 3980.774673] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3980.780078] Lustre: Skipped 1 previous similar message [ 3980.788930] Lustre: Skipped 1 previous similar message [ 3983.080392] Lustre: server umount lustre-OST0000 complete [ 3985.890183] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3985.896691] Lustre: Skipped 1 previous similar message [ 3999.199131] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3999.275860] Lustre: server umount lustre-MDT0000 complete [ 4001.022759] LustreError: 168088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786342411 with bad export cookie 3069753323124838383 [ 4001.025444] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4001.029794] LustreError: 168088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4001.182525] Lustre: server umount lustre-MDT0001 complete [ 4005.275906] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 02:13:35 (1786342415) [ 4012.717413] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 4016.913396] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 4033.495781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4033.628300] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4033.645888] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4033.699301] Lustre: lustre-MDT0000: new disk, initializing [ 4033.737732] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4033.744888] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4035.708949] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4041.298658] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4041.367358] Lustre: 172838:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4041.505285] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4043.445852] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4048.197812] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4050.615875] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4053.764033] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4053.910066] Lustre: 174814:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4056.703885] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4060.192244] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4060.268136] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4061.775380] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4065.249522] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4065.249880] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4065.254853] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4065.261898] Lustre: Skipped 1 previous similar message [ 4065.270100] Lustre: Skipped 7 previous similar messages [ 4067.133288] Lustre: server umount lustre-OST0000 complete [ 4071.042773] Lustre: server umount lustre-MDT0000 complete [ 4072.869832] LustreError: 173637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786342483 with bad export cookie 3069753323124839251 [ 4072.874517] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4072.877498] LustreError: 173637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4073.032256] Lustre: server umount lustre-MDT0001 complete [ 4077.160476] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 02:14:47 (1786342487) [ 4084.929373] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 4089.455206] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 4105.920236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4106.028284] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4106.041690] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4106.044397] Lustre: Skipped 3 previous similar messages [ 4106.082712] Lustre: lustre-MDT0000: new disk, initializing [ 4106.084335] Lustre: Skipped 2 previous similar messages [ 4106.122260] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4106.126378] Lustre: Skipped 2 previous similar messages [ 4106.134577] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4106.139296] Lustre: Skipped 2 previous similar messages [ 4107.893061] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4112.998501] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4113.053325] Lustre: 179125:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4113.168548] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4114.916862] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4119.755086] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4122.357202] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4125.231116] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4125.356672] Lustre: 181102:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4127.875699] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4131.897426] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4132.526044] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4136.930897] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4136.931146] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4136.935175] LustreError: Skipped 1 previous similar message [ 4136.940333] Lustre: Skipped 2 previous similar messages [ 4136.940979] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4136.950039] Lustre: Skipped 3 previous similar messages [ 4141.080463] Lustre: server umount lustre-OST0000 complete [ 4144.096181] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4148.840367] Lustre: server umount lustre-MDT0000 complete [ 4150.418355] LustreError: 179925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786342560 with bad export cookie 3069753323124840322 [ 4150.421587] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4150.427430] LustreError: 179925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4150.551643] Lustre: server umount lustre-MDT0001 complete [ 4154.687179] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 4155.885661] Key type lgssc unregistered [ 4156.048362] LNet: 182956:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4156.051249] LNetError: 182956:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4156.060314] LNet: Removed LNI 192.168.203.123@tcp [ 4156.438338] Key type .llcrypt unregistered [ 4156.439653] Key type ._llcrypt unregistered [ 4166.932921] Key type ._llcrypt registered [ 4166.934432] Key type .llcrypt registered [ 4166.987941] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 4167.340844] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4167.353228] alg: No test for adler32 (adler32-zlib) [ 4168.256923] Lustre: Lustre: Build Version: 2.17.56_50_g8cc021f [ 4168.381053] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 4169.983166] Key type lgssc registered [ 4170.419183] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4173.590180] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4173.602421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4174.816867] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4176.786749] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4180.673276] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4180.858168] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4182.797133] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4186.168580] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4188.736882] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4191.728668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4191.891265] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4194.419705] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4198.080176] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4200.108237] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4220.896890] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4220.900368] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4220.906216] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4221.923784] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4221.924605] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4221.927220] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4226.384316] Lustre: server umount lustre-OST0000 complete [ 4232.160771] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4232.161044] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4232.161416] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4232.161420] Lustre: Skipped 1 previous similar message [ 4232.165370] Lustre: Skipped 1 previous similar message [ 4237.280552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4237.284523] Lustre: Skipped 2 previous similar messages [ 4242.399094] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4242.401355] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4242.406670] Lustre: Skipped 1 previous similar message [ 4242.468535] Lustre: server umount lustre-MDT0000 complete [ 4243.938501] LustreError: 184329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786342654 with bad export cookie 18426739709080179324 [ 4243.941560] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4243.945088] LustreError: 184329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4244.095056] Lustre: server umount lustre-MDT0001 complete [ 4248.738214] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 4250.192564] Key type lgssc unregistered [ 4250.340788] LNet: 187907:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4250.344868] LNetError: 187907:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4250.357572] LNet: Removed LNI 192.168.203.123@tcp [ 4250.750533] Key type .llcrypt unregistered [ 4250.752456] Key type ._llcrypt unregistered [ 4261.728860] Key type ._llcrypt registered [ 4261.732102] Key type .llcrypt registered [ 4261.791196] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 4262.236621] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4262.250561] alg: No test for adler32 (adler32-zlib) [ 4263.131381] Lustre: Lustre: Build Version: 2.17.56_50_g8cc021f [ 4263.235538] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 4264.823136] Key type lgssc registered [ 4265.284195] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4268.636579] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4268.646281] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4269.853040] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4271.629909] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4275.396782] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4275.584627] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4277.529254] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4281.058023] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4283.354725] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4286.167365] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4286.300211] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4288.629659] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4292.066593] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4294.503257] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 4305.889809] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4305.895291] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4305.895874] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4305.901572] Lustre: Skipped 1 previous similar message [ 4306.966729] Lustre: server umount lustre-OST0000 complete [ 4310.262510] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9944454c5000.ost_server_uuid 50 [ 4325.647349] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 4336.292874] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4336.308401] Lustre: Skipped 1 previous similar message [ 4336.312857] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4336.316512] Lustre: Skipped 1 previous similar message [ 4341.729047] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4341.735084] Lustre: Skipped 1 previous similar message [ 4342.460715] Lustre: server umount lustre-MDT0000 complete [ 4344.248220] LustreError: 189282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786342754 with bad export cookie 8631636345015304699 [ 4344.252776] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4344.255591] LustreError: 189282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4344.388447] Lustre: server umount lustre-MDT0001 complete [ 4348.203380] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 02:19:18 (1786342758) [ 4350.932332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4351.141464] LustreError: 193233:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4351.152563] LustreError: 193233:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4351.180352] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4352.799886] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4356.189960] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4356.347328] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4358.058576] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4360.936783] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4363.196524] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4365.838945] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4365.980944] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4368.208806] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4371.133490] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4373.924702] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4374.018218] Lustre: lustre-OST0001: new disk, initializing [ 4374.020853] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4374.024870] Lustre: 196071:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4376.105261] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4376.426898] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4377.556125] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4377.561496] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4377.600730] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4381.054201] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4385.761289] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4385.764803] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4385.769685] Lustre: Skipped 1 previous similar message [ 4385.772372] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4385.775830] Lustre: Skipped 1 previous similar message [ 4388.225807] Lustre: server umount lustre-OST0000 complete [ 4390.440311] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4390.522629] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 4390.884205] LustreError: 196363:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4390.890466] LustreError: 196363:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4392.112811] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4392.200340] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 4396.001177] LustreError: 195053:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4396.007865] LustreError: 195053:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4401.123153] LustreError: 195053:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4401.129515] LustreError: 195053:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4403.168274] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4403.173922] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4403.182334] Lustre: Skipped 1 previous similar message [ 4403.185904] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4403.189551] Lustre: Skipped 2 previous similar messages [ 4406.242061] LustreError: 195054:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4406.251718] LustreError: 195054:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4408.468553] Lustre: server umount lustre-OST0001 complete [ 4411.360771] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4411.362243] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4411.366671] Lustre: Skipped 2 previous similar messages [ 4411.368663] Lustre: Skipped 2 previous similar messages [ 4416.161327] Lustre: server umount lustre-MDT0000 complete [ 4416.481293] LustreError: 193234:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4416.488511] LustreError: 193234:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4417.616346] LustreError: 193213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786342828 with bad export cookie 8631636345015305770 [ 4417.618909] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4417.620443] LustreError: 193213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4417.753206] Lustre: server umount lustre-MDT0001 complete [ 4427.141836] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 02:20:37 (1786342837) [ 4429.886259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4430.021498] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4430.045824] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4430.155293] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4430.157788] Lustre: Skipped 1 previous similar message [ 4431.871293] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4435.353464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4435.408318] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4435.432837] Lustre: 199483:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4435.467026] LustreError: 199493:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4437.264706] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4440.382987] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4442.605934] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4445.132265] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4445.216142] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4447.507963] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4450.327399] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4451.863630] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 4452.994279] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4453.032746] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4455.350364] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4458.131783] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4460.533730] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4460.539319] Lustre: Skipped 1 previous similar message [ 4460.539544] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4460.545020] Lustre: Skipped 1 previous similar message [ 4460.568083] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 4465.214054] Lustre: server umount lustre-OST0000 complete [ 4468.499058] LustreError: 202173:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4468.505368] LustreError: 202173:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4480.993255] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4480.994187] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4480.997149] LustreError: Skipped 2 previous similar messages [ 4482.079600] Lustre: server umount lustre-OST0001 complete [ 4489.625841] Lustre: server umount lustre-MDT0000 complete [ 4490.843131] LustreError: 199474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786342901 with bad export cookie 8631636345015307156 [ 4490.843638] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4490.847501] LustreError: 199474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4490.979467] Lustre: server umount lustre-MDT0001 complete [ 4499.185884] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 02:21:49 (1786342909) [ 4504.530903] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 4507.799049] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 4520.251922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4520.336187] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4520.349417] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4520.386609] Lustre: lustre-MDT0000: new disk, initializing [ 4520.406777] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4520.409144] Lustre: Skipped 3 previous similar messages [ 4520.414327] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4521.760249] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4526.063504] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4526.107284] Lustre: 207544:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4526.130195] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4526.132055] Lustre: Skipped 1 previous similar message [ 4526.169435] Lustre: lustre-MDT0001: new disk, initializing [ 4526.218874] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4526.224641] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4527.662798] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4531.737675] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4533.907777] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4536.326845] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4536.439201] Lustre: lustre-OST0000: new disk, initializing [ 4536.441952] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4536.445973] Lustre: 209514:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4538.547988] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4542.932439] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4545.151505] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4546.043570] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4546.049076] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4546.085894] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4546.277167] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 4547.954061] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4548.049449] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4551.139275] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4551.140935] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4551.143289] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4551.147523] Lustre: Skipped 4 previous similar messages [ 4551.151343] Lustre: Skipped 5 previous similar messages [ 4554.910607] Lustre: server umount lustre-OST0000 complete [ 4562.653885] Lustre: server umount lustre-MDT0000 complete [ 4564.257054] LustreError: 207536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786342974 with bad export cookie 8631636345015308913 [ 4564.260612] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4564.261076] LustreError: 207536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4564.417718] Lustre: server umount lustre-MDT0001 complete [ 4568.032495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4568.217867] LustreError: 211731:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4568.230154] LustreError: 211731:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 4570.066873] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4573.687843] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4575.523510] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4578.699972] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4580.799221] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4583.397970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4585.880648] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4588.976266] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4590.752564] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4592.903635] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 4594.709138] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4594.796568] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4598.240557] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4598.243745] LustreError: Skipped 2 previous similar messages [ 4601.685348] Lustre: server umount lustre-OST0000 complete [ 4604.326822] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4606.227691] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4630.271569] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4630.394027] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4630.397618] Lustre: Skipped 5 previous similar messages [ 4632.530244] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4635.399647] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4636.073864] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4637.385131] Lustre: server umount lustre-OST0000 complete [ 4640.736235] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4640.742354] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4640.751481] Lustre: Skipped 4 previous similar messages [ 4640.756089] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4640.760069] Lustre: Skipped 6 previous similar messages [ 4647.287884] Lustre: server umount lustre-MDT0000 complete [ 4648.927397] LustreError: 211711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786343059 with bad export cookie 8631636345015309781 [ 4648.930937] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4648.935174] LustreError: 211711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4649.079896] Lustre: server umount lustre-MDT0001 complete [ 4652.667720] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 02:24:23 (1786343063) [ 4655.189812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4656.961138] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4660.175266] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4661.935404] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4665.036386] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4667.403548] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4670.261839] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4672.726838] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4675.510677] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4677.186490] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4681.400487] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 4683.298938] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4685.446551] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 2 sec [ 4688.257304] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4688.338781] Lustre: lustre-OST0001: new disk, initializing [ 4688.341470] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4688.345339] Lustre: 221112:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4690.705946] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4695.002451] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4695.563345] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4695.569557] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4695.605113] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4696.788422] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4696.902561] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4698.581689] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4698.677575] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4700.643304] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4700.647139] LustreError: Skipped 1 previous similar message [ 4701.521717] Lustre: server umount lustre-OST0001 complete [ 4704.079849] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4704.167929] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 4705.765872] LustreError: 219523:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4705.772485] LustreError: 219523:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 4705.789859] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4705.875678] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 4729.786900] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4732.133678] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4734.895662] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4734.950779] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4734.956689] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4734.960249] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4734.965174] Lustre: Skipped 1 previous similar message [ 4735.436388] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4738.233550] Lustre: server umount lustre-OST0001 complete [ 4756.560767] Lustre: server umount lustre-OST0000 complete [ 4770.784478] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4770.787614] Lustre: Skipped 14 previous similar messages [ 4772.319171] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4772.417446] Lustre: server umount lustre-MDT0000 complete [ 4774.067543] LustreError: 217680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786343184 with bad export cookie 8631636345015310712 [ 4774.070604] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4774.074076] LustreError: 217680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4780.298513] Lustre: server umount lustre-MDT0001 complete [ 4789.193196] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 02:26:39 (1786343199) [ 4791.616070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4791.714895] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4791.726452] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4791.797854] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4791.800154] Lustre: Skipped 5 previous similar messages [ 4793.266164] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4796.302421] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4796.341717] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4796.353675] Lustre: 226342:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4797.949462] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4800.545309] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4802.631399] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4805.275765] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4805.349873] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4807.686409] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4810.380695] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4814.697347] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4814.727595] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4816.979229] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4819.606061] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4821.177080] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec [ 4822.749593] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4822.830049] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4824.394193] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4824.477708] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4826.519286] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9944454c7800.ost_server_uuid 50 [ 4827.134613] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9944454c7800.ost_server_uuid in FULL state after 0 sec [ 4827.559461] Lustre: Permanently deactivating lustre-OST0001 [ 4827.565827] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 4828.533188] Lustre: Permanently reactivating lustre-OST0001 [ 4830.695076] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4830.696251] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4830.700606] Lustre: Skipped 9 previous similar messages [ 4830.704115] LustreError: Skipped 5 previous similar messages [ 4835.655645] Lustre: server umount lustre-OST0001 complete [ 4843.567626] Lustre: server umount lustre-OST0000 complete [ 4859.359097] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4859.429146] Lustre: server umount lustre-MDT0000 complete [ 4860.840213] LustreError: 229223:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786343271 with bad export cookie 8631636345015311986 [ 4860.843388] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4860.845916] LustreError: 229223:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4860.975729] Lustre: server umount lustre-MDT0001 complete [ 4869.347953] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 02:27:59 (1786343279) [ 4872.178720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4872.295483] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4872.308197] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4872.313547] Lustre: Skipped 5 previous similar messages [ 4873.887893] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4876.618628] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4876.652933] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4876.664119] Lustre: 233038:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4878.085121] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4880.653089] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4882.556788] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4884.866918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4884.956362] Lustre: Permanently deactivating lustre-OST0000 [ 4884.962650] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 4887.146248] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4889.805351] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4892.316413] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4893.137256] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4893.141602] LustreError: 236023:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0001: cannot cleanup orphans: rc = -5 [ 4893.191776] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 4894.605701] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4897.155695] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4897.720615] Lustre: Permanently reactivating lustre-OST0000 [ 4902.392934] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 4902.396897] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4902.402379] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4902.417668] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 4902.425636] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 4911.802974] Lustre: server umount lustre-OST0001 complete [ 4919.189146] Lustre: server umount lustre-OST0000 complete [ 4926.748294] Lustre: server umount lustre-MDT0000 complete [ 4928.081565] LustreError: 233780:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786343338 with bad export cookie 8631636345015313617 [ 4928.085729] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4928.086326] LustreError: 233780:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4928.224267] Lustre: server umount lustre-MDT0001 complete [ 4931.788660] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 4932.882885] Key type lgssc unregistered [ 4933.014438] LNet: 237887:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4933.017353] LNetError: 237887:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4933.029472] LNet: Removed LNI 192.168.203.123@tcp [ 4933.365257] Key type .llcrypt unregistered [ 4933.367129] Key type ._llcrypt unregistered [ 4942.004844] Key type ._llcrypt registered [ 4942.005861] Key type .llcrypt registered [ 4942.046227] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 4942.453922] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4942.474594] alg: No test for adler32 (adler32-zlib) [ 4943.330885] Lustre: Lustre: Build Version: 2.17.56_50_g8cc021f [ 4943.415686] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 4944.999166] Key type lgssc registered [ 4945.374686] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4948.700658] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4948.706455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4949.848848] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4951.203314] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4954.256269] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4954.310759] Lustre: Found index 1 for lustre-MDT0001, updating log [ 4954.323209] Lustre: 239320:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4954.329022] Lustre: Permanently deactivating lustre-MDT0001 [ 4954.336304] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 4954.453994] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4955.977142] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4958.454180] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4958.702589] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 4958.705938] LustreError: 239363:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4958.710374] LustreError: 239363:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4958.714659] LustreError: 239363:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 1, failed: rc = -108 [ 4960.324563] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4962.726721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4962.837323] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4963.875105] LustreError: 241170:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4963.877436] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 4963.882337] LustreError: 241170:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4964.728227] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4967.167769] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4968.930031] LustreError: 241171:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4969.425483] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4969.510923] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4971.509681] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4974.062252] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4974.565280] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 4975.590819] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 4975.590845] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 4979.929316] Lustre: Permanently reactivating lustre-MDT0001 [ 4979.934933] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 4979.938493] Lustre: Skipped 2 previous similar messages [ 4982.680764] Lustre: lustre-MDT0001: Client 3312f02a-6887-4e04-af7a-50f702a9846c (at 192.168.203.23@tcp) reconnecting [ 4984.810816] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4985.313308] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4986.980360] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 4987.052958] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 4988.385239] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 4988.450996] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 4989.226133] Lustre: Permanently deactivating lustre-MDT0001 [ 4989.231909] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 4989.233653] Lustre: Skipped 2 previous similar messages [ 4991.360165] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 4994.529948] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 4994.532618] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4994.537098] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4996.064918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4996.065448] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4996.067222] Lustre: Skipped 1 previous similar message [ 4996.073394] Lustre: Skipped 1 previous similar message [ 4999.834979] Lustre: server umount lustre-MDT0000 complete [ 5001.026070] LustreError: 239310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786343411 with bad export cookie 4940810392954133911 [ 5001.030240] LustreError: 239310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5001.030739] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5001.136648] Lustre: server umount lustre-MDT0001 complete [ 5008.443698] Lustre: server umount lustre-OST0000 complete [ 5012.233204] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 02:30:22 (1786343422) [ 5015.993038] Lustre: server umount lustre-OST0001 complete [ 5018.096613] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 5021.333489] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 5033.560565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5033.645093] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5033.649683] Lustre: Skipped 2 previous similar messages [ 5033.660606] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5033.698567] Lustre: lustre-MDT0000: new disk, initializing [ 5033.723900] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5033.731715] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5035.104196] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5039.510298] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5039.559635] Lustre: 247573:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5039.597682] LustreError: 247585:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5039.602633] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5039.607573] Lustre: Skipped 1 previous similar message [ 5039.648802] Lustre: lustre-MDT0001: new disk, initializing [ 5039.699093] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5039.702746] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5041.194157] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5044.971594] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5046.782421] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5049.096422] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5049.194503] Lustre: lustre-OST0000: new disk, initializing [ 5049.196532] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5049.199787] Lustre: 249542:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5049.229466] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5049.231828] Lustre: Skipped 1 previous similar message [ 5051.172407] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5054.463504] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5054.468536] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5054.506729] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5055.210191] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5055.877173] LustreError: 247582:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 5060.241427] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5060.305267] Lustre: lustre-OST0001: new disk, initializing [ 5060.307668] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5060.311223] Lustre: 250656:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5062.287609] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5065.967165] LustreError: 247582:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 5065.979517] LustreError: 247581:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 5066.124916] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5066.717165] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5066.766209] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5066.772267] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5066.824216] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5076.071109] LustreError: 247581:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 5076.960680] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5076.961508] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5076.964629] LustreError: Skipped 1 previous similar message [ 5076.968744] Lustre: Skipped 1 previous similar message [ 5076.971280] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5076.974455] Lustre: Skipped 3 previous similar messages [ 5082.084665] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5082.087294] Lustre: Skipped 2 previous similar messages [ 5082.818349] Lustre: server umount lustre-OST0001 complete [ 5082.900735] LustreError: 250214:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5087.202333] LustreError: 251511:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5087.208850] LustreError: 251511:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5095.392192] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5095.392342] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5095.395285] LustreError: Skipped 1 previous similar message [ 5095.395907] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5095.399389] Lustre: Skipped 1 previous similar message [ 5097.445680] LustreError: 249524:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5097.452226] LustreError: 249524:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5107.682510] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5107.685743] Lustre: Skipped 6 previous similar messages [ 5109.215105] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5109.274167] Lustre: server umount lustre-OST0000 complete [ 5111.263820] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5111.267649] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5116.829854] Lustre: server umount lustre-MDT0000 complete [ 5117.922699] LustreError: 247586:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5117.929506] LustreError: 247586:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 5118.136626] LustreError: 247567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786343528 with bad export cookie 4940810392954136725 [ 5118.141969] LustreError: 247567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5118.142514] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5124.338916] Lustre: server umount lustre-MDT0001 complete [ 5128.090588] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 5129.222976] Key type lgssc unregistered [ 5129.359591] LNet: 252691:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5129.362503] LNetError: 252691:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5129.369431] LNet: Removed LNI 192.168.203.123@tcp [ 5129.702378] Key type .llcrypt unregistered [ 5129.703798] Key type ._llcrypt unregistered [ 5144.098527] Key type ._llcrypt registered [ 5144.099785] Key type .llcrypt registered [ 5144.137644] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 5144.454537] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5144.506812] alg: No test for adler32 (adler32-zlib) [ 5145.357126] Lustre: Lustre: Build Version: 2.17.56_50_g8cc021f [ 5145.442000] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 5147.023121] Key type lgssc registered [ 5147.364486] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5150.047461] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5150.055379] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5151.143207] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5151.151680] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5151.206376] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5152.580112] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5155.548528] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5155.597815] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5155.610912] Lustre: 254876:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5155.718033] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5157.160219] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5159.830367] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5161.902493] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5164.385162] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5164.452906] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5164.510547] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5166.428383] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5168.890941] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5173.268777] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 5214.176588] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5214.178717] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5214.179544] LustreError: Skipped 1 previous similar message [ 5214.184256] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5214.187213] Lustre: Skipped 1 previous similar message [ 5218.719396] Lustre: server umount lustre-OST0000 complete [ 5220.159586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 5224.225503] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5224.359761] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5226.524238] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5228.993799] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5233.572481] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5233.579252] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5233.579751] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5233.587124] Lustre: Skipped 1 previous similar message [ 5264.354610] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5264.354642] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5264.357102] LustreError: Skipped 1 previous similar message [ 5264.360840] Lustre: Skipped 1 previous similar message [ 5264.361573] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5264.366528] Lustre: Skipped 1 previous similar message [ 5268.440062] Lustre: server umount lustre-OST0000 complete [ 5273.568119] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5273.572098] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5273.577409] Lustre: Skipped 1 previous similar message [ 5273.579879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5275.877557] Lustre: server umount lustre-MDT0000 complete [ 5277.304780] LustreError: 254865:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786343687 with bad export cookie 6997519918995472746 [ 5277.309114] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5277.309642] LustreError: 254865:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5277.434943] Lustre: server umount lustre-MDT0001 complete [ 5281.714424] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 5282.880276] Key type lgssc unregistered [ 5283.029699] LNet: 260292:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5283.033562] LNetError: 260292:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5283.042574] LNet: Removed LNI 192.168.203.123@tcp [ 5283.382495] Key type .llcrypt unregistered [ 5283.383919] Key type ._llcrypt unregistered [ 5292.760328] Key type ._llcrypt registered [ 5292.762363] Key type .llcrypt registered [ 5292.801565] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 5293.237862] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5293.246315] alg: No test for adler32 (adler32-zlib) [ 5294.114122] Lustre: Lustre: Build Version: 2.17.56_50_g8cc021f [ 5294.202825] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 5295.791153] Key type lgssc registered [ 5296.197076] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5299.048822] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5299.056172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5300.199597] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5301.582411] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5304.500196] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5304.633939] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5306.131958] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5308.729595] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5310.688106] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5313.023633] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5313.153162] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5315.120282] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5317.599631] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5324.454497] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 5336.031826] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5336.035565] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5336.042311] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5340.130763] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5340.131885] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5340.132298] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5340.138778] Lustre: Skipped 1 previous similar message [ 5341.455325] Lustre: server umount lustre-OST0000 complete [ 5345.248791] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5345.249471] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5345.252388] Lustre: Skipped 1 previous similar message [ 5350.368128] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5350.370717] Lustre: Skipped 2 previous similar messages [ 5355.487939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5355.490143] Lustre: Skipped 1 previous similar message [ 5357.023137] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5357.079797] Lustre: server umount lustre-MDT0000 complete [ 5358.299471] LustreError: 261664:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786343768 with bad export cookie 14695087364007909429 [ 5358.303074] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5358.303969] LustreError: 261664:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5358.424478] Lustre: server umount lustre-MDT0001 complete [ 5362.110374] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 5363.174108] Key type lgssc unregistered [ 5363.299472] LNet: 266008:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5363.302484] LNetError: 266008:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5363.312334] LNet: Removed LNI 192.168.203.123@tcp [ 5363.642210] Key type .llcrypt unregistered [ 5363.643936] Key type ._llcrypt unregistered [ 5371.005413] Key type ._llcrypt registered [ 5371.006804] Key type .llcrypt registered [ 5371.054786] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 5371.384663] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5371.406080] alg: No test for adler32 (adler32-zlib) [ 5372.267780] Lustre: Lustre: Build Version: 2.17.56_50_g8cc021f [ 5372.366796] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 5373.951216] Key type lgssc registered [ 5374.317866] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5377.062903] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5377.070703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5378.230282] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5379.629134] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5382.513216] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5382.636756] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5383.991236] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5386.401347] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5388.186611] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5390.377651] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5390.494809] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5392.505611] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5394.993610] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5402.794118] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5413.346763] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5413.347930] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5413.349451] LustreError: Skipped 1 previous similar message [ 5413.350208] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5413.353457] Lustre: Skipped 1 previous similar message [ 5415.117078] Lustre: server umount lustre-OST0000 complete [ 5418.464993] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5418.465081] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5418.473131] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5418.474974] Lustre: Skipped 2 previous similar messages [ 5422.419484] Lustre: server umount lustre-MDT0000 complete [ 5423.544904] LustreError: 267196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786343834 with bad export cookie 16652498086851462811 [ 5423.547344] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5423.549166] LustreError: 267196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5423.655844] Lustre: server umount lustre-MDT0001 complete [ 5427.035469] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 5428.068454] Key type lgssc unregistered [ 5428.192394] LNet: 270912:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5428.194655] LNetError: 270912:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5428.203247] LNet: Removed LNI 192.168.203.123@tcp [ 5428.512047] Key type .llcrypt unregistered [ 5428.513192] Key type ._llcrypt unregistered [ 5436.978385] Key type ._llcrypt registered [ 5436.979333] Key type .llcrypt registered [ 5437.013621] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 5437.304121] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5437.353101] alg: No test for adler32 (adler32-zlib) [ 5438.197162] Lustre: Lustre: Build Version: 2.17.56_50_g8cc021f [ 5438.284466] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 5439.863164] Key type lgssc registered [ 5440.198167] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5442.614903] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5442.620759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5443.750757] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5444.993659] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5447.636170] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5447.751745] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5449.074665] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5451.374586] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5453.123805] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5455.285322] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5455.396103] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5457.241284] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5459.557180] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5468.709389] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5477.855900] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5477.858518] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5477.863456] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5478.880794] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5478.880886] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5478.881690] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5483.280277] Lustre: server umount lustre-OST0000 complete [ 5488.607835] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5488.611151] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5488.617086] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5488.619636] Lustre: Skipped 1 previous similar message [ 5490.707968] Lustre: server umount lustre-MDT0000 complete [ 5491.954648] LustreError: 272286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786343902 with bad export cookie 16403729507606002322 [ 5491.957365] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5491.959383] LustreError: 272286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5492.089904] Lustre: server umount lustre-MDT0001 complete [ 5495.976869] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 5497.024740] Key type lgssc unregistered [ 5497.156529] LNet: 276630:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5497.161138] LNetError: 276630:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5497.169227] LNet: Removed LNI 192.168.203.123@tcp [ 5497.483920] Key type .llcrypt unregistered [ 5497.485929] Key type ._llcrypt unregistered [ 5504.725345] Key type ._llcrypt registered [ 5504.726646] Key type .llcrypt registered [ 5504.764642] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 5505.213967] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5505.228312] alg: No test for adler32 (adler32-zlib) [ 5506.089424] Lustre: Lustre: Build Version: 2.17.56_50_g8cc021f [ 5506.171786] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 5507.751124] Key type lgssc registered [ 5508.103760] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5510.785059] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5510.791538] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5511.984097] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5513.407346] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5516.125160] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5516.237176] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5517.425038] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5519.632932] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5521.310604] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5523.399714] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5523.505631] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5525.332462] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5527.810764] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5536.805412] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 5545.951838] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5545.955683] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5545.962389] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5546.977636] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5546.979083] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5546.979245] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5552.096317] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5552.098914] Lustre: Skipped 2 previous similar messages [ 5556.191109] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5556.243669] Lustre: server umount lustre-OST0000 complete [ 5562.336759] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5562.336913] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5562.337358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5562.345981] Lustre: Skipped 1 previous similar message [ 5563.672097] Lustre: server umount lustre-MDT0000 complete [ 5564.897222] LustreError: 277812:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786343975 with bad export cookie 12390506621139341058 [ 5564.899885] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5564.901612] LustreError: 277812:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5565.025301] Lustre: server umount lustre-MDT0001 complete [ 5568.497418] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 5569.427215] Key type lgssc unregistered [ 5569.542403] LNet: 281564:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5569.544764] LNetError: 281564:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5569.550287] LNet: Removed LNI 192.168.203.123@tcp [ 5569.843506] Key type .llcrypt unregistered [ 5569.844724] Key type ._llcrypt unregistered [ 5573.069177] Key type ._llcrypt registered [ 5573.070229] Key type .llcrypt registered [ 5573.104248] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 5578.144045] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 5583.799514] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 5584.161990] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5584.196695] alg: No test for adler32 (adler32-zlib) [ 5585.038407] Lustre: Lustre: Build Version: 2.17.56_50_g8cc021f [ 5585.118710] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 5586.695126] Key type lgssc registered [ 5587.002387] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5597.339176] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5597.346180] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5598.421949] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5598.431354] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5598.464718] Lustre: lustre-MDT0000: new disk, initializing [ 5598.483493] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5598.488889] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5599.663300] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5603.699574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5603.739324] Lustre: 285088:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5603.761501] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5603.763183] Lustre: Skipped 1 previous similar message [ 5603.797434] Lustre: lustre-MDT0001: new disk, initializing [ 5603.826926] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5603.838226] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5603.841803] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5605.121897] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5608.563102] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5610.206903] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5612.324178] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5612.422697] Lustre: lustre-OST0000: new disk, initializing [ 5612.424247] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5612.426648] Lustre: 287055:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5612.452725] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5614.372183] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5617.927450] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5618.678294] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5618.681334] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5618.716420] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5619.269578] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5619.347288] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5620.661295] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5620.735674] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5623.779518] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5623.780319] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5623.780611] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5623.783279] Lustre: Skipped 1 previous similar message [ 5623.785408] LustreError: Skipped 1 previous similar message [ 5627.473400] Lustre: server umount lustre-OST0000 complete [ 5628.896227] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5628.896725] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5628.900213] Lustre: Skipped 1 previous similar message [ 5628.901957] Lustre: Skipped 1 previous similar message [ 5634.016244] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5634.019119] Lustre: Skipped 2 previous similar messages [ 5634.775577] Lustre: server umount lustre-MDT0000 complete [ 5635.900927] LustreError: 285079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786344046 with bad export cookie 13024370826730536340 [ 5635.903770] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5635.904438] LustreError: 285079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5636.011152] Lustre: server umount lustre-MDT0001 complete [ 5638.488928] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 02:40:49 (1786344049) [ 5640.641450] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5640.835967] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5642.170981] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5644.845593] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5644.952433] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5646.200346] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5648.393541] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5650.075470] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5652.164497] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5654.105110] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5656.462572] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5670.880432] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5670.882249] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5670.883174] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5670.885678] Lustre: Skipped 1 previous similar message [ 5675.784394] Lustre: server umount lustre-OST0000 complete [ 5680.097145] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 5680.100043] LustreError: Skipped 1 previous similar message [ 5680.101859] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5680.106640] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5680.108800] Lustre: Skipped 1 previous similar message [ 5691.359094] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5691.360100] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5691.364466] Lustre: Skipped 5 previous similar messages [ 5691.412073] Lustre: server umount lustre-MDT0000 complete [ 5692.534030] LustreError: 289443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786344103 with bad export cookie 13024370826730537194 [ 5692.536908] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5692.539389] LustreError: 289443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5692.648794] Lustre: server umount lustre-MDT0001 complete [ 5695.791445] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 5696.732279] Key type lgssc unregistered [ 5696.852351] LNet: 293018:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5696.854794] LNetError: 293018:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5696.864275] LNet: Removed LNI 192.168.203.123@tcp [ 5697.157057] Key type .llcrypt unregistered [ 5697.158522] Key type ._llcrypt unregistered [ 5704.853512] Key type ._llcrypt registered [ 5704.854520] Key type .llcrypt registered [ 5704.889383] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 5710.540415] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 5710.824960] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5710.876658] alg: No test for adler32 (adler32-zlib) [ 5711.717762] Lustre: Lustre: Build Version: 2.17.56_50_g8cc021f [ 5711.798832] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 5713.375145] Key type lgssc registered [ 5713.715519] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5727.450912] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5727.456827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5728.548554] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5728.558405] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5728.591650] Lustre: lustre-MDT0000: new disk, initializing [ 5728.614426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5728.620610] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5729.999290] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5734.031234] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5734.074416] Lustre: 296812:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5734.095804] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5734.097402] Lustre: Skipped 1 previous similar message [ 5734.131380] Lustre: lustre-MDT0001: new disk, initializing [ 5734.165137] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5734.177470] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5734.181125] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5735.474183] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5738.978061] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5740.554644] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5742.738859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5742.865252] Lustre: lustre-OST2710: new disk, initializing [ 5742.868359] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 5742.872614] Lustre: 298780:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5742.907711] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 5744.248585] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 5744.251866] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 5744.285578] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 5744.882589] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5748.602606] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5750.724746] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5750.781434] Lustre: lustre-OST03e8: new disk, initializing [ 5750.783413] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 5750.786280] Lustre: 299844:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5750.813704] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 5752.628653] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5754.821809] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5757.356380] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 5757.424173] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5757.952587] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 5757.957346] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 5757.983719] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 5758.664687] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 5758.733428] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5760.050213] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 5760.117061] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5761.399053] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 5761.464281] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5763.040555] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5763.043728] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5763.044982] Lustre: Skipped 3 previous similar messages [ 5763.046809] Lustre: Skipped 2 previous similar messages [ 5768.160107] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5768.162288] Lustre: Skipped 4 previous similar messages [ 5768.470641] Lustre: server umount lustre-MDT0000 complete [ 5769.657337] LustreError: 299843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786344180 with bad export cookie 819850894811606206 [ 5769.660677] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5769.661029] LustreError: 299843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5769.775377] Lustre: server umount lustre-MDT0001 complete [ 5780.480731] Lustre: server umount lustre-OST2710 complete [ 5791.826719] Lustre: server umount lustre-OST03e8 complete [ 5797.087857] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 5799.766955] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 5811.214724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5811.301020] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5811.312550] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5811.350119] Lustre: lustre-MDT0000: new disk, initializing [ 5811.370616] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5811.376797] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5812.729181] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5817.022738] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5817.062057] Lustre: 305082:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5817.151502] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5818.370306] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5821.926276] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5823.608470] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5825.658704] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5825.743932] Lustre: lustre-OST0000: new disk, initializing [ 5825.745637] Lustre: Skipped 1 previous similar message [ 5825.747474] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5825.749739] Lustre: Skipped 2 previous similar messages [ 5825.751872] Lustre: 307049:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5825.776820] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5825.778727] Lustre: Skipped 1 previous similar message [ 5827.561346] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5829.503449] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5829.506347] Lustre: Skipped 1 previous similar message [ 5829.508376] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5829.535827] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5831.265754] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5832.589880] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5832.653534] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5833.892277] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5833.968308] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5834.721269] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5834.721829] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5834.723700] LustreError: Skipped 1 previous similar message [ 5834.724232] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5834.728086] Lustre: Skipped 1 previous similar message [ 5839.841889] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5839.844065] Lustre: Skipped 2 previous similar messages [ 5840.720786] Lustre: server umount lustre-OST0000 complete [ 5842.911891] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5842.914860] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5844.960157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5844.960939] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5844.962955] Lustre: Skipped 1 previous similar message [ 5855.200308] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5855.202117] Lustre: Skipped 3 previous similar messages [ 5856.223207] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5856.285532] Lustre: server umount lustre-MDT0000 complete [ 5857.638174] LustreError: 307048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786344268 with bad export cookie 819850894811607865 [ 5857.642090] LustreError: 307048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5857.642101] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5857.775379] Lustre: server umount lustre-MDT0001 complete [ 5860.600223] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 02:44:31 (1786344271) [ 5861.085807] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 5861.591802] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 02:44:32 (1786344272) [ 5862.482780] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 5870.580020] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5870.654648] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5870.662826] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5870.713127] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5871.946639] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5874.615878] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5874.645578] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5874.654312] Lustre: 310901:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5876.007408] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5891.681645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5891.740955] Lustre: Denying initial registration attempt from nid 192.168.203.123@tcp, specified as failover [ 5891.743993] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 5891.746854] LustreError: 312350:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 5891.749602] LustreError: 312350:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 5891.751596] LustreError: 312350:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 5891.772621] Lustre: server umount lustre-OST0000 complete [ 5891.773988] LustreError: 312350:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 5895.135668] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5895.137598] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5895.140857] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5899.919164] Lustre: server umount lustre-MDT0000 complete [ 5901.034169] LustreError: 311639:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786344311 with bad export cookie 819850894811608733 [ 5901.037077] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5901.038489] LustreError: 311639:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5901.140950] Lustre: server umount lustre-MDT0001 complete [ 5904.294865] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 5905.169687] Key type lgssc unregistered [ 5905.290359] LNet: 313473:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5905.293503] LNetError: 313473:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5905.302259] LNet: Removed LNI 192.168.203.123@tcp [ 5905.594457] Key type .llcrypt unregistered [ 5905.595398] Key type ._llcrypt unregistered [ 5908.859098] Key type ._llcrypt registered [ 5908.860134] Key type .llcrypt registered [ 5908.897710] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 5909.186786] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5909.192715] alg: No test for adler32 (adler32-zlib) [ 5910.047993] Lustre: Lustre: Build Version: 2.17.56_50_g8cc021f [ 5910.125670] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 5911.703181] Key type lgssc registered [ 5912.031085] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5934.560701] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 5937.160646] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5937.165926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5938.245027] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5938.251566] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5938.300714] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5939.532742] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5942.138166] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5942.171721] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5942.182644] Lustre: 316210:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5942.276251] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5943.508291] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5945.731319] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5947.368852] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5962.530211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5962.590086] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5962.647073] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5964.376429] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5966.594409] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5967.288690] Lustre: server umount lustre-OST0000 complete [ 5971.948844] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5971.949840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5971.952424] Lustre: Skipped 1 previous similar message [ 5974.550223] Lustre: server umount lustre-MDT0000 complete [ 5975.671210] LustreError: 316202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786344386 with bad export cookie 6615174649480153573 [ 5975.674831] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5975.676033] LustreError: 316202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5975.785191] Lustre: server umount lustre-MDT0001 complete [ 5979.114867] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 5980.034749] Key type lgssc unregistered [ 5980.148424] LNet: 319774:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5980.151978] LNetError: 319774:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5980.164261] LNet: Removed LNI 192.168.203.123@tcp [ 5980.435752] Key type .llcrypt unregistered [ 5980.437301] Key type ._llcrypt unregistered [ 5988.742480] Key type ._llcrypt registered [ 5988.743394] Key type .llcrypt registered [ 5988.776176] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 5989.164607] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5989.175189] alg: No test for adler32 (adler32-zlib) [ 5990.022923] Lustre: Lustre: Build Version: 2.17.56_50_g8cc021f [ 5990.106342] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 5991.687152] Key type lgssc registered [ 5992.025847] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5994.402130] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5994.407906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5995.533572] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5996.826590] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5999.434982] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5999.550929] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6000.778165] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6002.984595] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6004.560872] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6008.801661] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6008.900112] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6010.773951] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6013.171085] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6026.721653] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6026.723172] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6026.725285] Lustre: Skipped 2 previous similar messages [ 6026.729027] Lustre: Skipped 2 previous similar messages [ 6028.570907] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.23@tcp (stopping) [ 6031.643648] Lustre: server umount lustre-MDT0000 complete [ 6031.841063] LustreError: 323657:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6031.847174] LustreError: 323657:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 6032.848160] LustreError: 321147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786344443 with bad export cookie 6452989973622380084 [ 6032.851243] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6032.852166] LustreError: 321147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6033.683271] LustreError: 323172:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6033.690219] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.23@tcp (stopping) [ 6036.960123] LustreError: 322648:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6036.960340] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6036.972916] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6043.104350] LustreError: 321163:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6043.104742] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6043.110179] LustreError: 321163:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6043.112672] Lustre: Skipped 2 previous similar messages [ 6047.199157] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6047.280530] Lustre: server umount lustre-MDT0001 complete [ 6049.394306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6052.184144] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6057.439623] LustreError: 320479:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000004fca77c7 x1873117865599872/t0(0) o250->MGC192.168.203.123@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6057.561990] LustreError: 322996:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6057.567643] LustreError: 322996:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6057.592647] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6057.621374] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [ 6058.984540] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6061.816315] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6061.953465] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6063.315495] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6065.702434] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6073.315068] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6075.381975] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 9 sec [ 6077.137490] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6077.663505] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6082.528583] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6082.532481] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6082.538936] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6082.542023] Lustre: Skipped 1 previous similar message [ 6083.554903] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6093.279132] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6093.336397] Lustre: server umount lustre-OST0000 complete [ 6097.887724] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6097.890283] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6097.894219] Lustre: Skipped 1 previous similar message [ 6098.912106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6098.914240] Lustre: Skipped 6 previous similar messages [ 6100.759697] Lustre: server umount lustre-MDT0000 complete [ 6102.097517] LustreError: 321901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786344512 with bad export cookie 6452989973622387203 [ 6102.102393] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6102.103279] LustreError: 321901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6102.244121] Lustre: server umount lustre-MDT0001 complete [ 6105.870090] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 6106.931398] Key type lgssc unregistered [ 6107.056438] LNet: 327452:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6107.059120] LNetError: 327452:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6107.068466] LNet: Removed LNI 192.168.203.123@tcp [ 6107.396731] Key type .llcrypt unregistered [ 6107.397786] Key type ._llcrypt unregistered [ 6116.531624] Key type ._llcrypt registered [ 6116.533328] Key type .llcrypt registered [ 6116.581722] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 6117.102456] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6117.133547] alg: No test for adler32 (adler32-zlib) [ 6117.990896] Lustre: Lustre: Build Version: 2.17.56_50_g8cc021f [ 6118.094746] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 6119.679124] Key type lgssc registered [ 6120.052033] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6122.586616] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6122.593830] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6123.736051] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6125.077310] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6127.830489] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6127.958597] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6129.383358] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6131.836751] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6133.641848] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6137.760733] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6137.761534] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6137.764634] Lustre: Skipped 1 previous similar message [ 6140.817052] Lustre: server umount lustre-MDT0000 complete [ 6141.993083] LustreError: 328826:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786344552 with bad export cookie 828687241748693222 [ 6141.995707] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6141.996645] LustreError: 328826:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6142.109758] Lustre: server umount lustre-MDT0001 complete [ 6146.577045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6146.665389] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6146.674499] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6146.741196] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6147.998644] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6150.662142] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6150.694268] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6150.705789] Lustre: 331432:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6150.788658] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6152.015305] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6154.255057] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6155.910299] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6160.226156] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6160.284239] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 6160.287615] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 6160.291163] LustreError: 333314:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 6160.293284] LustreError: 333314:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 6160.294881] LustreError: 333314:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 6160.313071] Lustre: server umount lustre-OST0000 complete [ 6160.314469] LustreError: 333314:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 6164.834065] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6164.900373] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6164.957411] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6166.667629] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6168.777923] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6171.276277] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6171.303419] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6171.326380] Lustre: lustre-OST0001: new disk, initializing [ 6171.327900] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6171.330371] Lustre: 334846:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6172.416746] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6172.421512] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6172.447714] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [ 6172.468306] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6173.025950] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6175.215111] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6177.764875] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6177.765987] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6177.766992] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6177.767017] Lustre: Skipped 1 previous similar message [ 6177.771355] Lustre: Skipped 1 previous similar message [ 6177.773772] LustreError: Skipped 1 previous similar message [ 6178.239947] Lustre: server umount lustre-OST0001 complete [ 6182.883029] LustreError: 333781:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6182.883357] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6182.883410] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6182.884228] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6182.884232] Lustre: Skipped 1 previous similar message [ 6182.888604] LustreError: 333781:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6185.615922] Lustre: server umount lustre-OST0000 complete [ 6186.975756] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6186.977907] LustreError: Skipped 1 previous similar message [ 6186.979273] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6186.983497] Lustre: Skipped 1 previous similar message [ 6186.985267] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6186.987214] Lustre: Skipped 1 previous similar message [ 6193.120109] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6193.122424] Lustre: Skipped 3 previous similar messages [ 6201.311170] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6201.371346] Lustre: server umount lustre-MDT0000 complete [ 6202.585223] LustreError: 331467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786344613 with bad export cookie 828687241748693761 [ 6202.588447] LustreError: 331467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6202.589039] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6202.709385] Lustre: server umount lustre-MDT0001 complete [ 6206.216265] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 6207.209342] Key type lgssc unregistered [ 6207.327396] LNet: 336809:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6207.331224] LNetError: 336809:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6207.340324] LNet: Removed LNI 192.168.203.123@tcp [ 6207.645414] Key type .llcrypt unregistered [ 6207.647077] Key type ._llcrypt unregistered [ 6238.313237] Key type ._llcrypt registered [ 6238.314278] Key type .llcrypt registered [ 6238.351405] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 6244.543688] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 6244.929746] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6244.973085] alg: No test for adler32 (adler32-zlib) [ 6245.830641] Lustre: Lustre: Build Version: 2.17.56_50_g8cc021f [ 6245.927382] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 6247.503096] Key type lgssc registered [ 6247.872045] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6259.040780] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6259.046332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6260.128353] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6260.141523] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6260.176502] Lustre: lustre-MDT0000: new disk, initializing [ 6260.198824] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6260.206606] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6261.593788] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6265.623900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6265.668478] Lustre: 341724:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6265.697210] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6265.699283] Lustre: Skipped 1 previous similar message [ 6265.738735] Lustre: lustre-MDT0001: new disk, initializing [ 6265.777305] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6265.791609] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6265.796351] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6267.219905] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6270.887958] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6272.680596] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6274.939791] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6275.049630] Lustre: lustre-OST0000: new disk, initializing [ 6275.052234] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6275.056455] Lustre: 343691:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6275.091754] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6276.667107] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6276.670227] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6276.706527] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6276.970281] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6280.727953] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6282.171671] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6282.250147] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6283.671690] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6283.747453] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6286.482542] Lustre: server umount lustre-OST0000 complete [ 6291.423755] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6291.426671] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6291.431630] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6291.936484] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6291.941113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6297.056253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6297.058913] Lustre: Skipped 1 previous similar message [ 6302.175091] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6302.176134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6302.181837] Lustre: Skipped 1 previous similar message [ 6302.242224] Lustre: server umount lustre-MDT0000 complete [ 6303.199120] Lustre: 339954:0:(client.c:2506:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786344697/real 1786344697] req@000000005943a8e7 x1873118134036608/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786344713 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6303.209995] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6303.441164] LustreError: 342519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786344714 with bad export cookie 8557033075892708514 [ 6303.443462] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6303.445037] LustreError: 342519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6303.560393] Lustre: server umount lustre-MDT0001 complete [ 6306.257623] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 02:51:56 (1786344716) [ 6308.674468] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 02:51:59 (1786344719) [ 6311.490407] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6311.667411] LustreError: 346443:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6311.675020] LustreError: 346443:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6311.693282] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6313.137508] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6316.066363] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6316.193420] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6317.561285] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6320.039642] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6321.884384] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6324.048060] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6326.066430] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6328.447221] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6342.237462] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 6342.790457] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 6343.322399] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 6343.894473] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 6344.531157] Lustre: Failing over lustre-MDT0000 [ 6344.747107] Lustre: server umount lustre-MDT0000 complete [ 6347.231968] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6347.234266] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6347.238025] LustreError: 346443:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6349.086160] LustreError: 347875:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6349.093116] LustreError: 347875:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6352.864107] LustreError: 347169:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6352.872104] LustreError: 347169:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6357.740756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6357.782723] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6357.873918] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6357.877749] Lustre: Skipped 1 previous similar message [ 6357.903895] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6359.253878] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6359.315651] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6363.111647] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6363.121965] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 6363.144285] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6364.915460] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6365.452904] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6367.329159] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 6368.051384] Lustre: Failing over lustre-MDT0000 [ 6368.225697] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6368.226512] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6368.229787] Lustre: Skipped 3 previous similar messages [ 6368.234286] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6368.237631] Lustre: Skipped 1 previous similar message [ 6368.315381] Lustre: server umount lustre-MDT0000 complete [ 6369.563862] LustreError: 348931:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6369.571137] LustreError: 348931:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6378.464116] LustreError: 346443:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6378.469328] LustreError: 346443:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 6381.214266] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6381.256296] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6381.342204] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6381.371592] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6382.681924] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6384.915277] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6386.660212] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6386.662102] Lustre: Skipped 2 previous similar messages [ 6386.670265] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 6386.688112] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 6388.399830] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6388.899713] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6391.777174] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6391.777548] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6391.779480] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6391.789226] Lustre: Skipped 2 previous similar messages [ 6397.365178] Lustre: server umount lustre-MDT0000 complete [ 6399.003435] LustreError: 346425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786344809 with bad export cookie 8557033075892711195 [ 6399.010959] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6399.014573] LustreError: 346425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6399.177542] Lustre: server umount lustre-MDT0001 complete [ 6411.309201] Lustre: server umount lustre-OST0000 complete [ 6416.403909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6416.535821] LustreError: 352793:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6416.540572] LustreError: 352793:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6416.553832] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6417.764967] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6420.396692] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6421.819798] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6424.027175] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6425.731424] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6427.944483] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6429.788249] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6431.994694] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6442.409755] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 6451.679865] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6451.683064] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6451.689322] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6451.692165] Lustre: Skipped 5 previous similar messages [ 6454.669731] Lustre: server umount lustre-OST0000 complete [ 6462.040258] Lustre: server umount lustre-MDT0000 complete [ 6462.944063] LustreError: 352793:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6462.950663] LustreError: 352793:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6463.246047] LustreError: 352773:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786344873 with bad export cookie 8557033075892711916 [ 6463.248933] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6463.249829] LustreError: 352773:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6463.361590] Lustre: server umount lustre-MDT0001 complete [ 6466.705185] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 6467.737505] Key type lgssc unregistered [ 6467.866407] LNet: 356345:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6467.868527] LNetError: 356345:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6467.874269] LNet: Removed LNI 192.168.203.123@tcp [ 6468.167040] Key type .llcrypt unregistered [ 6468.168513] Key type ._llcrypt unregistered [ 6477.556299] Key type ._llcrypt registered [ 6477.557671] Key type .llcrypt registered [ 6477.594611] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 6477.951894] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6478.013407] alg: No test for adler32 (adler32-zlib) [ 6478.863218] Lustre: Lustre: Build Version: 2.17.56_50_g8cc021f [ 6478.947278] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 6480.519126] Key type lgssc registered [ 6480.887718] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6483.491197] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6483.497984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6484.652288] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6486.056218] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6489.075533] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6489.208274] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6490.610111] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6492.890158] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6494.606809] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6496.913727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6497.036083] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6499.175166] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6501.499682] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6511.401386] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 6516.705065] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6516.706321] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6516.709320] Lustre: Skipped 2 previous similar messages [ 6516.711682] Lustre: Skipped 1 previous similar message [ 6521.813963] Lustre: server umount lustre-MDT0000 complete [ 6521.824409] LustreError: 357837:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6521.829237] LustreError: 357837:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 6523.015430] LustreError: 357864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786344933 with bad export cookie 6088505975386594468 [ 6523.018765] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6523.019782] LustreError: 357864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6523.130986] Lustre: server umount lustre-MDT0001 complete [ 6533.594517] Lustre: server umount lustre-OST0000 complete [ 6537.832329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6537.972817] LustreError: 361568:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6537.995745] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6539.248525] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6542.010325] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6542.125753] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6543.460539] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6545.806709] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6547.625650] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6549.902732] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6551.967265] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6554.404340] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6564.393537] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 6573.023870] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6573.027619] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6573.033449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6573.036356] Lustre: Skipped 1 previous similar message [ 6574.561325] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6574.562299] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6574.566763] Lustre: Skipped 1 previous similar message [ 6574.569229] Lustre: Skipped 1 previous similar message [ 6578.070758] Lustre: server umount lustre-MDT0000 complete [ 6579.317960] LustreError: 361548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786344989 with bad export cookie 6088505975386595567 [ 6579.321641] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6579.322127] LustreError: 361548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6579.435629] Lustre: server umount lustre-MDT0001 complete [ 6590.929695] Lustre: server umount lustre-OST0000 complete [ 6595.560661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6595.702994] LustreError: 365495:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6595.722949] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6595.725022] Lustre: Skipped 1 previous similar message [ 6596.978761] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6599.728351] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6601.204553] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6603.546926] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6605.313635] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6607.563365] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6609.536586] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6611.847139] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6621.030697] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 6625.247843] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6625.250629] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6625.255250] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6625.258208] Lustre: Skipped 1 previous similar message [ 6626.274138] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6629.840939] Lustre: server umount lustre-OST0000 complete [ 6631.392284] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6631.392804] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6631.397641] Lustre: Skipped 2 previous similar messages [ 6631.400111] Lustre: Skipped 3 previous similar messages [ 6637.208773] Lustre: server umount lustre-MDT0000 complete [ 6638.454277] LustreError: 365477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786345049 with bad export cookie 6088505975386596575 [ 6638.457447] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6638.458670] LustreError: 365477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6638.571702] Lustre: server umount lustre-MDT0001 complete [ 6642.228869] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 6643.284313] Key type lgssc unregistered [ 6643.410377] LNet: 369049:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6643.413401] LNetError: 369049:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6643.423365] LNet: Removed LNI 192.168.203.123@tcp [ 6643.740943] Key type .llcrypt unregistered [ 6643.742457] Key type ._llcrypt unregistered [ 6653.338014] Key type ._llcrypt registered [ 6653.339226] Key type .llcrypt registered [ 6653.382823] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 6653.843427] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6653.857830] alg: No test for adler32 (adler32-zlib) [ 6654.729741] Lustre: Lustre: Build Version: 2.17.56_50_g8cc021f [ 6654.829591] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 6656.407142] Key type lgssc registered [ 6656.805382] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6659.410549] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6659.416809] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6659.420559] LustreError: 370466:0:(osd_handler.c:8554:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 6659.427748] LustreError: 370466:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 6659.430075] LustreError: 370466:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 6659.432704] LustreError: 370466:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 6659.436505] LustreError: 370466:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6661.956129] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6661.959196] LustreError: 370862:0:(osd_handler.c:8554:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 6661.968299] LustreError: 370862:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 6661.971073] LustreError: 370862:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 6661.973650] LustreError: 370862:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 6661.977054] LustreError: 370862:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6668.701792] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 6669.773587] Key type lgssc unregistered [ 6669.914507] LNet: 371866:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6669.917890] LNetError: 371866:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6669.926361] LNet: Removed LNI 192.168.203.123@tcp [ 6670.219446] Key type .llcrypt unregistered [ 6670.220895] Key type ._llcrypt unregistered [ 6675.992645] Key type ._llcrypt registered [ 6675.993591] Key type .llcrypt registered [ 6676.031541] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 6682.101834] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 6682.398066] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6682.406238] alg: No test for adler32 (adler32-zlib) [ 6683.258835] Lustre: Lustre: Build Version: 2.17.56_50_g8cc021f [ 6683.343503] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 6684.927131] Key type lgssc registered [ 6685.280243] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6696.369891] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6696.374566] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6697.449120] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6697.458287] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6697.493127] Lustre: lustre-MDT0000: new disk, initializing [ 6697.511835] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6697.518407] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6698.822434] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6702.920829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6702.961632] Lustre: 374993:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6702.984562] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6702.986455] Lustre: Skipped 1 previous similar message [ 6703.017202] Lustre: lustre-MDT0001: new disk, initializing [ 6703.045903] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6703.058380] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6703.062830] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6704.333561] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6708.093444] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6709.948700] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6712.122844] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6712.225796] Lustre: lustre-OST0000: new disk, initializing [ 6712.227500] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6712.229891] Lustre: 376961:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6712.255630] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6714.123206] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6714.742377] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6714.746071] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6714.776713] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6717.692109] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6719.017387] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6719.085426] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6720.368930] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6720.446243] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6725.091438] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6725.092043] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6725.093523] LustreError: Skipped 1 previous similar message [ 6725.100063] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6725.102267] Lustre: Skipped 1 previous similar message [ 6730.209018] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6730.210917] Lustre: Skipped 1 previous similar message [ 6735.327139] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6735.329333] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6735.332945] Lustre: Skipped 1 previous similar message [ 6735.378928] Lustre: server umount lustre-OST0000 complete [ 6738.911852] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6738.914072] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6738.918224] Lustre: Skipped 1 previous similar message [ 6738.920628] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6740.448843] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6742.741033] Lustre: server umount lustre-MDT0000 complete [ 6743.973204] LustreError: 375789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786345154 with bad export cookie 3651077520309401007 [ 6743.975946] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6743.977669] LustreError: 375789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6744.088835] Lustre: server umount lustre-MDT0001 complete [ 6746.814829] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 6747.310442] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 02:59:18 (1786345158) [ 6749.370615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6749.529898] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6750.882229] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6753.664572] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6753.794730] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6755.151532] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6757.504205] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6759.198045] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6761.284037] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6763.162629] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6765.414080] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6767.619200] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6767.692920] Lustre: lustre-OST0001: new disk, initializing [ 6767.695345] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6767.698919] Lustre: 382255:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6767.702390] Lustre: lustre-OST0001: Not available for connect from 0@lo (not set up) [ 6767.705324] Lustre: Skipped 2 previous similar messages [ 6767.741383] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6767.744685] Lustre: Skipped 1 previous similar message [ 6769.589973] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6773.150947] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6773.225500] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6773.229522] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6773.244566] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6775.974856] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6775.980603] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6775.983716] Lustre: Skipped 1 previous similar message [ 6781.050328] Lustre: server umount lustre-OST0001 complete [ 6781.412549] LustreError: 381237:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6781.418253] LustreError: 381237:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6786.529932] LustreError: 381800:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6786.537858] LustreError: 381800:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6791.651222] LustreError: 381800:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6791.657015] LustreError: 381800:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6795.231768] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6795.234114] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6795.238183] Lustre: Skipped 1 previous similar message [ 6795.239719] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6795.241353] Lustre: Skipped 1 previous similar message [ 6796.771165] LustreError: 381235:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6796.775547] LustreError: 381235:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6798.988909] Lustre: server umount lustre-OST0000 complete [ 6806.483749] Lustre: server umount lustre-MDT0000 complete [ 6807.009352] LustreError: 381718:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6807.015252] LustreError: 381718:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6807.736346] LustreError: 380153:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786345218 with bad export cookie 3651077520309401875 [ 6807.740696] LustreError: 380153:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6807.741947] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6822.367134] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6822.449510] Lustre: server umount lustre-MDT0001 complete [ 6825.811797] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 6826.714189] Key type lgssc unregistered [ 6826.824592] LNet: 384242:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6826.826598] LNetError: 384242:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6826.835259] LNet: Removed LNI 192.168.203.123@tcp [ 6827.150063] Key type .llcrypt unregistered [ 6827.151411] Key type ._llcrypt unregistered [ 6844.104268] Key type ._llcrypt registered [ 6844.105529] Key type .llcrypt registered [ 6844.140718] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 6844.419225] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6844.441067] alg: No test for adler32 (adler32-zlib) [ 6845.292538] Lustre: Lustre: Build Version: 2.17.56_50_g8cc021f [ 6845.381242] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 6846.959204] Key type lgssc registered [ 6847.302651] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6849.712089] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6849.719490] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6850.802909] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6850.812091] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6850.866913] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6852.232741] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6854.896556] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6854.930288] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6854.943326] Lustre: 387018:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6855.043896] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6856.343933] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6858.543482] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6860.364408] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6861.280471] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6861.281250] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6861.285249] Lustre: Skipped 1 previous similar message [ 6861.290540] Lustre: Skipped 1 previous similar message [ 6866.400460] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6866.402690] Lustre: Skipped 1 previous similar message [ 6867.219705] Lustre: server umount lustre-MDT0000 complete [ 6868.430359] LustreError: 387009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786345279 with bad export cookie 15958455845250128301 [ 6868.433375] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6868.434693] LustreError: 387009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6868.561522] Lustre: server umount lustre-MDT0001 complete [ 6871.451335] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 03:01:22 (1786345282) [ 6873.591762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6873.751500] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6875.133624] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6877.962865] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6878.067566] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6879.268083] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6881.517181] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6883.169725] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6885.281044] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6885.340876] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6885.392832] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6887.085598] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6889.229554] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6892.848778] Lustre: Permanently deactivating lustre-OST0000 [ 6892.853897] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6893.312042] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6894.681748] LustreError: 392376:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 6894.686461] LustreError: 392376:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [ 6894.689633] LustreError: 392376:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 6895.277750] LustreError: 392424:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [ 6895.280078] LustreError: 392424:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 6901.965810] Lustre: server umount lustre-OST0000 complete [ 6902.887526] LustreError: 392673:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [ 6902.890073] LustreError: 392673:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 6903.776803] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6903.776896] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6903.777514] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6903.786287] Lustre: Skipped 1 previous similar message [ 6908.896609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6908.899882] Lustre: Skipped 2 previous similar messages [ 6914.016458] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6914.019050] Lustre: Skipped 1 previous similar message [ 6914.041358] Lustre: server umount lustre-MDT0000 complete [ 6915.368387] LustreError: 389560:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786345325 with bad export cookie 15958455845250128868 [ 6915.372288] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6915.374159] LustreError: 389560:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6924.815780] Lustre: server umount lustre-MDT0001 complete [ 6927.613541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6928.925022] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6929.813539] LustreError: 393829:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 6931.831606] Lustre: 394167:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.123@tcp [ 6932.514657] Lustre: server umount MGS complete [ 6936.096044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6936.248247] LustreError: 394890:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6936.253732] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6936.254785] LustreError: 394890:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6936.257248] Lustre: Skipped 1 previous similar message [ 6936.276084] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6937.575234] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6940.174397] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6940.264351] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6941.548231] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6943.788142] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6945.442304] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6946.028266] Lustre: Permanently reactivating lustre-OST0000 [ 6946.033157] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6946.035967] Lustre: Skipped 2 previous similar messages [ 6966.239116] LustreError: 396467:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 6968.905873] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6969.028470] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6969.030411] Lustre: Skipped 1 previous similar message [ 6970.890418] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6973.271972] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6986.723099] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6986.723364] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6986.724278] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6986.725910] LustreError: Skipped 1 previous similar message [ 6986.732869] Lustre: Skipped 1 previous similar message [ 6987.283188] Lustre: server umount lustre-OST0000 complete [ 6990.614359] Lustre: server umount lustre-MDT0000 complete [ 6991.834650] LustreError: 394871:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786345402 with bad export cookie 15958455845250129890 [ 6991.839296] LustreError: 394871:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6991.839554] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6991.954576] Lustre: server umount lustre-MDT0001 complete [ 6995.531078] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 6996.595842] Key type lgssc unregistered [ 6996.726300] LNet: 399365:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6996.729545] LNetError: 399365:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6996.738343] LNet: Removed LNI 192.168.203.123@tcp [ 6997.041650] Key type .llcrypt unregistered [ 6997.042813] Key type ._llcrypt unregistered [ 7003.045615] Key type ._llcrypt registered [ 7003.046986] Key type .llcrypt registered [ 7003.089438] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 7009.008320] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 7009.287958] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7009.308827] alg: No test for adler32 (adler32-zlib) [ 7010.157387] Lustre: Lustre: Build Version: 2.17.56_50_g8cc021f [ 7010.236721] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 7011.815122] Key type lgssc registered [ 7012.144755] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7022.412692] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7022.418165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7023.501826] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7023.512789] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7023.549868] Lustre: lustre-MDT0000: new disk, initializing [ 7023.568681] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7023.575128] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7024.768293] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7028.758156] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7028.797956] Lustre: 402492:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7028.821688] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7028.823377] Lustre: Skipped 1 previous similar message [ 7028.855414] Lustre: lustre-MDT0001: new disk, initializing [ 7028.888792] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7028.901996] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7028.905040] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7030.108278] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7033.623434] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7035.397536] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7037.609694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7037.721037] Lustre: lustre-OST0000: new disk, initializing [ 7037.723666] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7037.728040] Lustre: 404459:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7037.758590] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7039.676811] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7043.297706] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7044.718773] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7046.136375] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7046.139890] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7046.180564] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7046.831406] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 7048.171906] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7048.234629] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7051.232774] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7051.234973] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7051.236029] LustreError: Skipped 1 previous similar message [ 7051.240307] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7051.244510] Lustre: Skipped 1 previous similar message [ 7054.991874] Lustre: server umount lustre-OST0000 complete [ 7056.352495] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7056.353280] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7056.357193] Lustre: Skipped 2 previous similar messages [ 7061.472306] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7061.475355] Lustre: Skipped 2 previous similar messages [ 7062.363677] Lustre: server umount lustre-MDT0000 complete [ 7063.600489] LustreError: 402486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786345474 with bad export cookie 16011359681253455249 [ 7063.603876] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7063.608575] LustreError: 402486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7063.742248] Lustre: server umount lustre-MDT0001 complete [ 7066.386270] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 7066.913394] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 03:04:37 (1786345477) [ 7069.117048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7069.298332] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7070.651266] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7073.475162] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7073.602582] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7074.954111] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7077.254882] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7079.051317] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7081.295387] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7083.292664] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7085.712796] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7086.355812] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 7097.184688] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 7098.849313] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7098.850300] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7098.851606] LustreError: Skipped 1 previous similar message [ 7098.855903] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7098.859679] Lustre: Skipped 1 previous similar message [ 7103.969516] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7103.972516] Lustre: Skipped 1 previous similar message [ 7104.718611] Lustre: server umount lustre-OST0000 complete [ 7109.089656] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7109.093200] Lustre: Skipped 2 previous similar messages [ 7114.208743] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7114.211368] Lustre: Skipped 3 previous similar messages [ 7120.351120] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7120.411431] Lustre: server umount lustre-MDT0000 complete [ 7121.546137] LustreError: 406908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786345532 with bad export cookie 16011359681253456117 [ 7121.550378] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7121.666155] Lustre: server umount lustre-MDT0001 complete [ 7125.222987] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 7126.247205] Key type lgssc unregistered [ 7126.367315] LNet: 410676:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7126.370525] LNetError: 410676:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7126.382304] LNet: Removed LNI 192.168.203.123@tcp [ 7126.668244] Key type .llcrypt unregistered [ 7126.669238] Key type ._llcrypt unregistered [ 7134.418717] Key type ._llcrypt registered [ 7134.419779] Key type .llcrypt registered [ 7134.457619] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 7134.572467] Key type .llcrypt unregistered [ 7134.573516] Key type ._llcrypt unregistered [ 7141.352338] Key type ._llcrypt registered [ 7141.353777] Key type .llcrypt registered [ 7141.390867] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 7141.732436] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7141.783506] alg: No test for adler32 (adler32-zlib) [ 7142.632927] Lustre: Lustre: Build Version: 2.17.56_50_g8cc021f [ 7142.719499] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 7144.303179] Key type lgssc registered [ 7144.659088] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7147.247754] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7147.253853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7148.393111] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7149.686344] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7152.406806] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7152.518960] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7154.434898] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7156.828713] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7157.664488] LustreError: 412868:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7157.669642] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 7157.672208] LustreError: 412868:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 7159.089398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7159.232491] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7160.664174] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7167.967931] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7167.972060] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7167.975957] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7169.508049] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7169.508076] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7169.510583] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7174.033799] Lustre: server umount lustre-OST0000 complete [ 7178.722540] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 7178.725255] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7178.729738] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7178.732510] Lustre: Skipped 1 previous similar message [ 7181.400823] Lustre: server umount lustre-MDT0000 complete [ 7182.620215] LustreError: 412851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786345593 with bad export cookie 17777136937188674596 [ 7182.624654] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7182.625574] LustreError: 412851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7182.744769] Lustre: server umount lustre-MDT0001 complete [ 7186.179400] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 7187.153813] Key type lgssc unregistered [ 7187.274334] LNet: 416018:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7187.276792] LNetError: 416018:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7187.284295] LNet: Removed LNI 192.168.203.123@tcp [ 7187.573257] Key type .llcrypt unregistered [ 7187.574329] Key type ._llcrypt unregistered [ 7196.363901] Key type ._llcrypt registered [ 7196.364992] Key type .llcrypt registered [ 7196.402247] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 7196.840305] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7196.848824] alg: No test for adler32 (adler32-zlib) [ 7197.690107] Lustre: Lustre: Build Version: 2.17.56_50_g8cc021f [ 7197.776765] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 7199.359141] Key type lgssc registered [ 7199.691323] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7202.178635] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7202.184557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7217.631302] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7222.751260] LustreError: 417384:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.123@tcp: failed processing log, type 4: rc = -110 [ 7249.375157] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7251.316592] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7253.684798] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7255.933148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7256.093563] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7256.118605] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 7257.350557] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7260.110892] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7260.231031] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7261.557455] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7263.981612] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7265.758517] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7269.859019] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7269.859653] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7269.862518] Lustre: Skipped 1 previous similar message [ 7269.865718] Lustre: Skipped 1 previous similar message [ 7274.381288] Lustre: server umount lustre-OST0000 complete [ 7279.076491] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 7279.080403] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7279.090261] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7280.096397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7280.097254] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7281.826404] Lustre: server umount lustre-MDT0000 complete [ 7284.269539] LustreError: 417392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786345694 with bad export cookie 8706867261373914928 [ 7284.269886] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7284.276555] LustreError: 417392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7284.465949] Lustre: server umount lustre-MDT0001 complete [ 7292.485280] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 7294.202901] Key type lgssc unregistered [ 7294.373625] LNet: 420956:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7294.378379] LNetError: 420956:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7294.387881] LNet: Removed LNI 192.168.203.123@tcp [ 7294.952369] Key type .llcrypt unregistered [ 7294.956628] Key type ._llcrypt unregistered [ 7309.083735] Key type ._llcrypt registered [ 7309.084555] Key type .llcrypt registered [ 7309.116840] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 7309.500774] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7309.547711] alg: No test for adler32 (adler32-zlib) [ 7310.417341] Lustre: Lustre: Build Version: 2.17.56_50_g8cc021f [ 7310.527572] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 7312.127128] Key type lgssc registered [ 7312.605238] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7315.182290] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7315.187308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7316.335274] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7317.581861] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7320.356663] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7320.472190] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7321.789763] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7326.226664] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7329.601580] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7332.760929] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7332.935964] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7335.460418] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7338.426401] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7341.160508] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 7346.144127] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7346.147264] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7346.153134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7346.657295] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7346.658822] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7346.664818] Lustre: Skipped 1 previous similar message [ 7346.669059] Lustre: Skipped 1 previous similar message [ 7347.680384] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7347.683284] Lustre: Skipped 1 previous similar message [ 7350.044422] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.23@tcp (stopping) [ 7351.960596] Lustre: server umount lustre-MDT0000 complete [ 7352.738747] LustreError: 422330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.23@tcp arrived at 1786345763 with bad export cookie 16823477415580874729 [ 7352.745916] LustreError: 422330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7352.801529] LustreError: 422349:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7352.810292] LustreError: 422349:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 7353.466248] LustreError: 422331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786345764 with bad export cookie 16823477415580874113 [ 7353.471445] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7353.471899] LustreError: 422331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7357.920472] LustreError: 423590:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7357.920619] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7357.927364] LustreError: 423590:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7357.933786] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7358.943891] LustreError: 422349:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7358.949014] LustreError: 422349:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7359.574857] Lustre: server umount lustre-OST0000 complete [ 7361.969212] LustreError: 423078:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000020403a4a x1873119250768768/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7362.035250] Lustre: server umount lustre-MDT0001 complete [ 7366.119018] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 7367.330163] Key type lgssc unregistered [ 7367.471595] LNet: 426034:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7367.475794] LNetError: 426034:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7367.486466] LNet: Removed LNI 192.168.203.123@tcp [ 7367.858215] Key type .llcrypt unregistered [ 7367.859937] Key type ._llcrypt unregistered [ 7377.995022] Key type ._llcrypt registered [ 7377.996305] Key type .llcrypt registered [ 7378.039633] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 7378.407149] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7378.450798] alg: No test for adler32 (adler32-zlib) [ 7379.344142] Lustre: Lustre: Build Version: 2.17.56_50_g8cc021f [ 7379.449547] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 7381.039169] Key type lgssc registered [ 7381.478704] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7384.587503] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7384.595287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7385.742826] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7387.119418] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7390.054087] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7390.189167] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7391.748182] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7394.356773] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7396.476179] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7399.100258] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7399.229414] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7401.455181] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7404.181751] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7410.536870] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 7415.778708] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 7415.779123] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7415.790624] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7415.793780] Lustre: Skipped 1 previous similar message [ 7419.676897] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.23@tcp (stopping) [ 7420.897746] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7420.900751] Lustre: Skipped 1 previous similar message [ 7421.777804] Lustre: server umount lustre-MDT0001 complete [ 7426.016113] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7426.016893] LustreError: 427431:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7426.017654] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7426.017659] Lustre: Skipped 1 previous similar message [ 7426.018164] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7426.037035] LustreError: 427431:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7429.723946] Lustre: server umount lustre-OST0000 complete [ 7431.137146] LustreError: 427449:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7431.348527] Lustre: server umount lustre-MDT0000 complete [ 7436.656353] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 7437.929292] Key type lgssc unregistered [ 7438.083210] LNet: 431138:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7438.088089] LNetError: 431138:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7438.102412] LNet: Removed LNI 192.168.203.123@tcp [ 7438.471421] Key type .llcrypt unregistered [ 7438.473134] Key type ._llcrypt unregistered [ 7448.227164] Key type ._llcrypt registered [ 7448.228733] Key type .llcrypt registered [ 7448.276117] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 7456.002369] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 7456.558150] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7456.574977] alg: No test for adler32 (adler32-zlib) [ 7457.465152] Lustre: Lustre: Build Version: 2.17.56_50_g8cc021f [ 7457.596029] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 7459.191153] Key type lgssc registered [ 7459.703706] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7477.391129] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 7484.082341] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7484.094364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7485.194921] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7485.213280] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7485.256185] Lustre: lustre-MDT0000: new disk, initializing [ 7485.291511] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7485.299283] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7487.020113] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7493.795721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7493.914797] Lustre: lustre-OST0000: new disk, initializing [ 7493.916950] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7493.919197] Lustre: Skipped 1 previous similar message [ 7493.922394] Lustre: 436142:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7493.952606] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7496.306883] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7500.066557] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7502.335082] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 7502.338874] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 7502.348671] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 7510.839587] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7510.881141] Lustre: 435128:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7510.909827] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7510.949298] Lustre: lustre-MDT0001: new disk, initializing [ 7510.987681] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7511.000974] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 7511.004597] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 7512.335303] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7515.817455] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7515.886590] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7517.150416] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7521.279754] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 4 sec [ 7525.345390] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7525.345860] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 7525.349257] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7525.352542] Lustre: Skipped 1 previous similar message [ 7527.708084] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.23@tcp (stopping) [ 7530.466141] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7530.468970] Lustre: Skipped 1 previous similar message [ 7532.819184] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.23@tcp (stopping) [ 7537.119134] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7537.185775] Lustre: server umount lustre-MDT0001 complete [ 7545.825694] LustreError: 436150:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7545.832631] LustreError: 436150:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 7550.944493] LustreError: 438473:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7550.944640] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7550.956886] Lustre: Skipped 1 previous similar message [ 7550.958586] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7550.960612] Lustre: Skipped 2 previous similar messages [ 7555.603107] Lustre: server umount lustre-OST0000 complete [ 7556.065551] LustreError: 435137:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7562.927738] Lustre: server umount lustre-MDT0000 complete [ 7567.114545] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 7568.114427] Key type lgssc unregistered [ 7568.234417] LNet: 439921:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7568.236816] LNetError: 439921:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7568.246746] LNet: Removed LNI 192.168.203.123@tcp [ 7568.567578] Key type .llcrypt unregistered [ 7568.568731] Key type ._llcrypt unregistered [ 7580.765484] Key type ._llcrypt registered [ 7580.766524] Key type .llcrypt registered [ 7580.804421] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 7580.905291] Key type .llcrypt unregistered [ 7580.906445] Key type ._llcrypt unregistered [ 7587.568396] Key type ._llcrypt registered [ 7587.569356] Key type .llcrypt registered [ 7587.607820] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 7587.974671] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7588.026538] alg: No test for adler32 (adler32-zlib) [ 7588.875885] Lustre: Lustre: Build Version: 2.17.56_50_g8cc021f [ 7588.972810] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 7590.551129] Key type lgssc registered [ 7590.912883] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7599.458803] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7599.465233] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7600.541833] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7600.550799] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7600.590234] Lustre: lustre-MDT0000: new disk, initializing [ 7600.609947] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7600.615122] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7601.932357] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7606.008689] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7606.057893] Lustre: 443285:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7606.082907] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7606.085360] Lustre: Skipped 1 previous similar message [ 7606.120346] Lustre: lustre-MDT0001: new disk, initializing [ 7606.151613] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7606.164858] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7606.170179] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7607.594614] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7611.307028] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7613.177132] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7615.407168] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7615.515474] Lustre: lustre-OST0000: new disk, initializing [ 7615.516993] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7615.519375] Lustre: 445253:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7615.548192] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7617.146189] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7617.149944] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7617.176623] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7617.444470] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7621.070133] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7626.719805] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7626.723118] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7626.729697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7627.744908] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7627.745278] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7627.749561] Lustre: Skipped 1 previous similar message [ 7627.752874] Lustre: Skipped 2 previous similar messages [ 7629.539717] Lustre: server umount lustre-MDT0000 complete [ 7630.833292] LustreError: 445252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786346041 with bad export cookie 18037677781385758341 [ 7630.835700] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7630.836973] LustreError: 445252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7630.956251] Lustre: server umount lustre-MDT0001 complete [ 7642.457044] Lustre: server umount lustre-OST0000 complete [ 7645.956060] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 03:14:16 (1786346056) [ 7648.608985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7648.779669] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7650.161694] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7653.176941] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7653.303248] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7654.679415] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7657.191319] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7659.001468] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7674.209332] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7674.274118] Lustre: Found index 0 for lustre-OST0000, updating log [ 7674.329451] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7675.365993] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 7676.114429] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7678.314429] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7680.479895] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7680.482621] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7680.487132] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7683.553708] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7683.557440] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7683.563099] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7685.527092] Lustre: server umount lustre-OST0000 complete [ 7698.108166] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7698.108851] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7698.113736] Lustre: Skipped 1 previous similar message [ 7698.117097] Lustre: Skipped 3 previous similar messages [ 7704.282925] Lustre: server umount lustre-MDT0000 complete [ 7705.533041] LustreError: 447257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786346116 with bad export cookie 18037677781385759720 [ 7705.536353] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7705.537948] LustreError: 447257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7705.657511] Lustre: server umount lustre-MDT0001 complete [ 7708.359483] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 03:15:19 (1786346119) [ 7737.439224] LNet: 1 local NIs in recovery (showing 1): 192.168.203.123@tcp [ 8287.425957] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8287.574542] LustreError: 453680:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8287.582350] LustreError: 453680:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8287.596749] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8288.893014] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 8291.655432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8293.104933] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 8295.492022] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8297.265066] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8312.417781] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8312.500651] Lustre: Found index 0 for lustre-OST0000, updating log [ 8314.402609] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 8314.662058] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [ 8316.703252] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8318.369325] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8318.373829] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8318.375620] Lustre: Skipped 2 previous similar messages [ 8323.473874] Lustre: server umount lustre-OST0000 complete [ 8325.664942] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8325.666286] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8325.669185] Lustre: Skipped 1 previous similar message [ 8325.673610] Lustre: Skipped 1 previous similar message [ 8327.712202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8327.714239] Lustre: Skipped 3 previous similar messages [ 8330.775726] Lustre: server umount lustre-MDT0000 complete [ 8330.785197] LustreError: 454407:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8330.793162] LustreError: 454407:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8331.808087] LustreError: 455500:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8331.812857] LustreError: 455500:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8331.980196] LustreError: 454414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786346742 with bad export cookie 18037677781385760791 [ 8331.982335] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8331.983650] LustreError: 454414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8332.096093] Lustre: server umount lustre-MDT0001 complete [ 8338.527338] LNet: 1 local NIs in recovery (showing 1): 192.168.203.123@tcp [ 8349.190799] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 8352.176109] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 8363.317672] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 03:26:13 (1786346773) [ 8368.892765] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 8369.956766] Key type lgssc unregistered [ 8375.199502] LNet: 460331:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8375.202400] LNetError: 460331:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8443.813317] LNet: Removed LNI 192.168.203.123@tcp [ 8444.124551] Key type .llcrypt unregistered [ 8444.126136] Key type ._llcrypt unregistered [ 8450.508419] Key type ._llcrypt registered [ 8450.509692] Key type .llcrypt registered [ 8450.545495] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 8450.959077] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8450.967687] alg: No test for adler32 (adler32-zlib) [ 8451.809500] Lustre: Lustre: Build Version: 2.17.56_50_g8cc021f [ 8451.899306] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 8453.479086] Key type lgssc registered [ 8453.830405] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8455.204970] LNet: Added LNI 192.168.203.123@tcp100 [8/256/0/180] [ 8455.488253] LNet: Added LNI 192.168.203.123@tcp101 [8/256/0/180] [ 8455.764163] LNet: Added LNI 192.168.203.123@tcp102 [8/256/0/180] [ 8456.035712] LNet: Added LNI 192.168.203.123@tcp103 [8/256/0/180] [ 8456.315873] LNet: Added LNI 192.168.203.123@tcp104 [8/256/0/180] [ 8456.590145] LNet: Added LNI 192.168.203.123@tcp105 [8/256/0/180] [ 8456.883359] LNet: Added LNI 192.168.203.123@tcp106 [8/256/0/180] [ 8457.194345] LNet: Added LNI 192.168.203.123@tcp107 [8/256/0/180] [ 8457.475727] LNet: Added LNI 192.168.203.123@tcp108 [8/256/0/180] [ 8457.753196] LNet: Added LNI 192.168.203.123@tcp109 [8/256/0/180] [ 8458.036415] LNet: Added LNI 192.168.203.123@tcp110 [8/256/0/180] [ 8458.317914] LNet: Added LNI 192.168.203.123@tcp111 [8/256/0/180] [ 8458.615221] LNet: Added LNI 192.168.203.123@tcp112 [8/256/0/180] [ 8458.902407] LNet: Added LNI 192.168.203.123@tcp113 [8/256/0/180] [ 8459.186281] LNet: Added LNI 192.168.203.123@tcp114 [8/256/0/180] [ 8459.458710] LNet: Added LNI 192.168.203.123@tcp115 [8/256/0/180] [ 8459.742572] LNet: Added LNI 192.168.203.123@tcp116 [8/256/0/180] [ 8460.024993] LNet: Added LNI 192.168.203.123@tcp117 [8/256/0/180] [ 8460.302982] LNet: Added LNI 192.168.203.123@tcp118 [8/256/0/180] [ 8460.579963] LNet: Added LNI 192.168.203.123@tcp119 [8/256/0/180] [ 8460.861179] LNet: Added LNI 192.168.203.123@tcp120 [8/256/0/180] [ 8461.155293] LNet: Added LNI 192.168.203.123@tcp121 [8/256/0/180] [ 8461.437950] LNet: Added LNI 192.168.203.123@tcp122 [8/256/0/180] [ 8461.726431] LNet: Added LNI 192.168.203.123@tcp123 [8/256/0/180] [ 8462.015442] LNet: Added LNI 192.168.203.123@tcp124 [8/256/0/180] [ 8462.305212] LNet: Added LNI 192.168.203.123@tcp125 [8/256/0/180] [ 8462.593354] LNet: Added LNI 192.168.203.123@tcp126 [8/256/0/180] [ 8462.881548] LNet: Added LNI 192.168.203.123@tcp127 [8/256/0/180] [ 8463.187152] LNet: Added LNI 192.168.203.123@tcp128 [8/256/0/180] [ 8463.485951] LNet: Added LNI 192.168.203.123@tcp129 [8/256/0/180] [ 8463.771304] LNet: Added LNI 192.168.203.123@tcp130 [8/256/0/180] [ 8464.064982] LNet: Added LNI 192.168.203.123@tcp131 [8/256/0/180] [ 8464.360639] LNet: Added LNI 192.168.203.123@tcp132 [8/256/0/180] [ 8464.647944] LNet: Added LNI 192.168.203.123@tcp133 [8/256/0/180] [ 8464.928225] LNet: Added LNI 192.168.203.123@tcp134 [8/256/0/180] [ 8465.216903] LNet: Added LNI 192.168.203.123@tcp135 [8/256/0/180] [ 8467.356344] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8467.365057] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8468.400511] Lustre: 463329:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.123@tcp: can't add NID 192.168.203.123@tcp130: rc = -75 [ 8468.426304] LNetError: 463349:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.123 [ 8468.433739] LNetError: Protocol error connecting to 192.168.203.123@tcp112 at host 192.168.203.123:988: is it running a compatible version of Lustre? [ 8468.437536] LNetError: Skipped 5 previous similar messages [ 8468.480227] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8468.489939] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8468.525289] Lustre: lustre-MDT0000: new disk, initializing [ 8468.546710] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8468.552335] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8469.343185] LNet: 3 local NIs in recovery (showing 3): 192.168.203.123@tcp124, 192.168.203.123@tcp125, 192.168.203.123@tcp123 [ 8469.345994] LNetError: 463356:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.123 [ 8469.352480] LNetError: 463356:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 6 previous similar messages [ 8469.362510] LNetError: Protocol error connecting to 192.168.203.123@tcp104 at host 192.168.203.123:988: is it running a compatible version of Lustre? [ 8469.368328] LNetError: Skipped 1 previous similar message [ 8469.865608] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 8473.864522] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8473.918658] Lustre: 463346:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8473.947972] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8473.950097] Lustre: Skipped 1 previous similar message [ 8473.990383] Lustre: lustre-MDT0001: new disk, initializing [ 8474.001741] Lustre: 464148:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.123@tcp: can't add NID 192.168.203.123@tcp130: rc = -75 [ 8474.004703] Lustre: 464148:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [ 8474.043673] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8474.062037] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8474.065764] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8475.335416] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 8476.913454] Lustre: 464513:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.123@tcp: can't add NID 192.168.203.123@tcp130: rc = -75 [ 8476.916552] Lustre: 464513:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [ 8478.917806] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8480.636981] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8482.771691] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8482.884278] Lustre: lustre-OST0000: new disk, initializing [ 8482.886274] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8482.888934] Lustre: 465319:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8482.918644] Lustre: 465324:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.123@tcp: can't add NID 192.168.203.123@tcp130: rc = -75 [ 8482.921274] Lustre: 465324:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [ 8482.937191] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8484.228511] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8484.233841] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8484.270985] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8484.800849] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 8488.332091] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8494.562163] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8494.563605] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8494.566196] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8494.569652] Lustre: Skipped 1 previous similar message [ 8494.575198] Lustre: Skipped 1 previous similar message [ 8496.655122] Lustre: server umount lustre-OST0000 complete [ 8499.039855] LNetError: 460792:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.123 [ 8499.043025] LNetError: 460792:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 1 previous similar message [ 8499.045379] LNetError: Protocol error connecting to 192.168.203.123@tcp119 at host 192.168.203.123:988: is it running a compatible version of Lustre? [ 8499.680496] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8499.681587] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8499.682153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8499.682719] LustreError: Skipped 1 previous similar message [ 8499.690901] Lustre: Skipped 1 previous similar message [ 8503.137226] LNetError: 463357:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.123 [ 8503.144700] LNetError: Protocol error connecting to 192.168.203.123@tcp112 at host 192.168.203.123:988: is it running a compatible version of Lustre? [ 8503.960226] Lustre: server umount lustre-MDT0000 complete [ 8505.103068] LustreError: 463337:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786346915 with bad export cookie 9496238356952970516 [ 8505.106186] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8505.107260] LustreError: 463337:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8505.225508] Lustre: server umount lustre-MDT0001 complete [ 8508.573479] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 8509.280025] LNetError: 460793:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.123 [ 8509.285375] LNetError: 460793:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 2 previous similar messages [ 8509.290271] LNetError: Protocol error connecting to 192.168.203.123@tcp121 at host 192.168.203.123:988: is it running a compatible version of Lustre? [ 8509.294052] LNetError: Skipped 2 previous similar messages [ 8509.553728] Key type lgssc unregistered [ 8509.700457] LNet: 467233:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8509.704578] LNetError: 467233:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8509.708074] LNet: Removed LNI 192.168.203.123@tcp [ 8510.751113] LNet: Removed LNI 192.168.203.123@tcp100 [ 8510.753102] LNetError: 467233:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8510.756515] LNetError: 467233:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 8510.760343] LNet: Removed LNI 192.168.203.123@tcp101 [ 8511.775306] LNet: Removed LNI 192.168.203.123@tcp102 [ 8511.778374] LNetError: 467233:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8511.783168] LNetError: 467233:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 8511.786031] LNet: Removed LNI 192.168.203.123@tcp103 [ 8512.799201] LNet: Removed LNI 192.168.203.123@tcp104 [ 8513.823227] LNet: Removed LNI 192.168.203.123@tcp105 [ 8513.825606] LNetError: 467233:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8513.830093] LNetError: 467233:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 8513.834198] LNet: Removed LNI 192.168.203.123@tcp106 [ 8514.847164] LNet: Removed LNI 192.168.203.123@tcp107 [ 8515.871180] LNet: Removed LNI 192.168.203.123@tcp108 [ 8516.895176] LNet: Removed LNI 192.168.203.123@tcp109 [ 8517.919169] LNet: Removed LNI 192.168.203.123@tcp110 [ 8517.921107] LNetError: 467233:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8517.925016] LNetError: 467233:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 8517.929955] LNet: Removed LNI 192.168.203.123@tcp111 [ 8518.943095] LNet: Removed LNI 192.168.203.123@tcp112 [ 8519.967131] LNet: Removed LNI 192.168.203.123@tcp113 [ 8520.991154] LNet: Removed LNI 192.168.203.123@tcp114 [ 8522.015146] LNet: Removed LNI 192.168.203.123@tcp115 [ 8523.039189] LNet: Removed LNI 192.168.203.123@tcp116 [ 8524.063187] LNet: Removed LNI 192.168.203.123@tcp117 [ 8525.087178] LNet: Removed LNI 192.168.203.123@tcp118 [ 8526.111156] LNet: Removed LNI 192.168.203.123@tcp119 [ 8526.113602] LNetError: 467233:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8526.116372] LNetError: 467233:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [ 8526.119299] LNet: Removed LNI 192.168.203.123@tcp120 [ 8527.135140] LNet: Removed LNI 192.168.203.123@tcp121 [ 8528.159125] LNet: Removed LNI 192.168.203.123@tcp122 [ 8529.183131] LNet: Removed LNI 192.168.203.123@tcp123 [ 8530.207156] LNet: Removed LNI 192.168.203.123@tcp124 [ 8531.231139] LNet: Removed LNI 192.168.203.123@tcp125 [ 8531.232744] LNet: Removed LNI 192.168.203.123@tcp126 [ 8531.233892] LNet: Removed LNI 192.168.203.123@tcp127 [ 8531.235359] LNet: Removed LNI 192.168.203.123@tcp128 [ 8531.236724] LNet: Removed LNI 192.168.203.123@tcp129 [ 8531.237855] LNet: Removed LNI 192.168.203.123@tcp130 [ 8531.238981] LNet: Removed LNI 192.168.203.123@tcp131 [ 8531.240183] LNet: Removed LNI 192.168.203.123@tcp132 [ 8531.241322] LNet: Removed LNI 192.168.203.123@tcp133 [ 8531.242895] LNet: Removed LNI 192.168.203.123@tcp134 [ 8531.250489] LNet: Removed LNI 192.168.203.123@tcp135 [ 8531.514218] Key type .llcrypt unregistered [ 8531.515354] Key type ._llcrypt unregistered [ 8538.675895] Key type ._llcrypt registered [ 8538.676919] Key type .llcrypt registered [ 8538.713640] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 8544.558661] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 8544.967548] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8544.974652] alg: No test for adler32 (adler32-zlib) [ 8545.820244] Lustre: Lustre: Build Version: 2.17.56_50_g8cc021f [ 8545.910566] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 8547.487113] Key type lgssc registered [ 8547.838483] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8558.535564] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 03:29:29 (1786346969) [ 8564.132446] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 8565.092590] Key type lgssc unregistered [ 8565.218375] LNet: 471296:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8565.221168] LNetError: 471296:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8565.230307] LNet: Removed LNI 192.168.203.123@tcp [ 8565.515511] Key type .llcrypt unregistered [ 8565.516700] Key type ._llcrypt unregistered [ 8571.908504] Key type ._llcrypt registered [ 8571.909511] Key type .llcrypt registered [ 8571.948566] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 8572.375942] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8572.381935] alg: No test for adler32 (adler32-zlib) [ 8573.231054] Lustre: Lustre: Build Version: 2.17.56_50_g8cc021f [ 8573.314564] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 8574.887216] Key type lgssc registered [ 8575.238091] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8578.156101] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8578.162212] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8579.241964] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8579.251367] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8579.287256] Lustre: lustre-MDT0000: new disk, initializing [ 8579.306163] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8579.311612] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8580.711355] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 8584.726485] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8584.769343] Lustre: 472486:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8584.791912] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8584.794013] Lustre: Skipped 1 previous similar message [ 8584.833124] Lustre: lustre-MDT0001: new disk, initializing [ 8584.862717] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8584.894377] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8584.899832] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8586.179706] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 8589.737249] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8591.507885] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8593.716833] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8593.822388] Lustre: lustre-OST0000: new disk, initializing [ 8593.823990] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8593.826544] Lustre: 474450:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8593.856559] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8595.683912] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 8599.262912] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8600.429467] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing lnet_if_list [ 8601.596326] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8601.600352] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8601.641901] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8602.001363] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [ 8602.012638] LNet: Added LNI 192.168.203.123@tcp10 [8/256/0/180] [ 8603.733750] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [ 8603.743035] LNet: Added LNI 192.168.203.123@tcp11 [8/256/0/180] [ 8605.012295] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [ 8605.024198] LNet: Added LNI 192.168.203.123@tcp12 [8/256/0/180] [ 8606.331323] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [ 8606.342830] LNet: Added LNI 192.168.203.123@tcp13 [8/256/0/180] [ 8607.666650] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [ 8607.678601] LNet: Added LNI 192.168.203.123@tcp14 [8/256/0/180] [ 8609.040553] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [ 8609.051849] LNet: Added LNI 192.168.203.123@tcp15 [8/256/0/180] [ 8610.420620] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [ 8610.431327] LNet: Added LNI 192.168.203.123@tcp16 [8/256/0/180] [ 8611.825688] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [ 8611.836833] LNet: Added LNI 192.168.203.123@tcp17 [8/256/0/180] [ 8613.280728] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [ 8613.293818] LNet: Added LNI 192.168.203.123@tcp18 [8/256/0/180] [ 8614.789746] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [ 8614.800643] LNet: Added LNI 192.168.203.123@tcp19 [8/256/0/180] [ 8616.303556] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [ 8616.313618] LNet: Added LNI 192.168.203.123@tcp20 [8/256/0/180] [ 8618.083662] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [ 8618.094894] LNet: Added LNI 192.168.203.123@tcp21 [8/256/0/180] [ 8619.694240] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [ 8619.710216] LNet: Added LNI 192.168.203.123@tcp22 [8/256/0/180] [ 8621.352564] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [ 8621.361416] LNet: Added LNI 192.168.203.123@tcp23 [8/256/0/180] [ 8622.975172] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [ 8622.985706] LNet: Added LNI 192.168.203.123@tcp24 [8/256/0/180] [ 8624.620123] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [ 8624.631147] LNet: Added LNI 192.168.203.123@tcp25 [8/256/0/180] [ 8626.337511] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [ 8626.347925] LNet: Added LNI 192.168.203.123@tcp26 [8/256/0/180] [ 8628.177526] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [ 8628.195254] LNet: Added LNI 192.168.203.123@tcp27 [8/256/0/180] [ 8630.096181] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [ 8630.106962] LNet: Added LNI 192.168.203.123@tcp28 [8/256/0/180] [ 8631.954867] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [ 8631.964911] LNet: Added LNI 192.168.203.123@tcp29 [8/256/0/180] [ 8633.817742] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [ 8633.827704] LNet: Added LNI 192.168.203.123@tcp30 [8/256/0/180] [ 8635.727304] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [ 8635.737774] LNet: Added LNI 192.168.203.123@tcp31 [8/256/0/180] [ 8637.640757] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [ 8637.649929] LNet: Added LNI 192.168.203.123@tcp32 [8/256/0/180] [ 8639.558082] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [ 8639.567536] LNet: Added LNI 192.168.203.123@tcp33 [8/256/0/180] [ 8641.505000] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [ 8641.514203] LNet: Added LNI 192.168.203.123@tcp34 [8/256/0/180] [ 8643.587430] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [ 8643.600334] LNet: Added LNI 192.168.203.123@tcp35 [8/256/0/180] [ 8645.788458] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [ 8645.798388] LNet: Added LNI 192.168.203.123@tcp36 [8/256/0/180] [ 8648.077898] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [ 8648.090539] LNet: Added LNI 192.168.203.123@tcp37 [8/256/0/180] [ 8650.273749] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [ 8650.285854] LNet: Added LNI 192.168.203.123@tcp38 [8/256/0/180] [ 8652.639333] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [ 8652.663495] LNet: Added LNI 192.168.203.123@tcp39 [8/256/0/180] [ 8655.739512] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [ 8655.758193] LNet: Added LNI 192.168.203.123@tcp40 [8/256/0/180] [ 8658.087471] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [ 8658.097934] LNet: Added LNI 192.168.203.123@tcp41 [8/256/0/180] [ 8660.567376] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 8660.584420] LNet: Added LNI 192.168.203.123@tcp42 [8/256/0/180] [ 8662.998853] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [ 8663.009845] LNet: Added LNI 192.168.203.123@tcp43 [8/256/0/180] [ 8665.300205] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [ 8665.309550] LNet: Added LNI 192.168.203.123@tcp44 [8/256/0/180] [ 8667.658731] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [ 8667.672413] LNet: Added LNI 192.168.203.123@tcp45 [8/256/0/180] [ 8670.154172] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [ 8670.165366] LNet: Added LNI 192.168.203.123@tcp46 [8/256/0/180] [ 8672.607783] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [ 8672.618790] LNet: Added LNI 192.168.203.123@tcp47 [8/256/0/180] [ 8675.061063] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [ 8675.072204] LNet: Added LNI 192.168.203.123@tcp48 [8/256/0/180] [ 8677.642502] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [ 8677.652221] LNet: Added LNI 192.168.203.123@tcp49 [8/256/0/180] [ 8680.242116] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [ 8680.251804] LNet: Added LNI 192.168.203.123@tcp50 [8/256/0/180] [ 8688.609623] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8688.614092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8688.616889] Lustre: Skipped 1 previous similar message [ 8693.728671] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8693.731845] Lustre: Skipped 4 previous similar messages [ 8694.674594] Lustre: server umount lustre-MDT0000 complete [ 8695.943518] LustreError: 473279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786347106 with bad export cookie 3888953285799153479 [ 8695.948217] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8695.949779] LustreError: 473279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8696.093070] Lustre: server umount lustre-MDT0001 complete [ 8703.472709] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 8706.526510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8715.231164] Lustre: 471758:0:(client.c:2506:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786347109/real 1786347109] req@00000000df957285 x1873120574126976/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786347125 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8715.243252] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8715.248723] Lustre: Skipped 1 previous similar message [ 8717.279104] Lustre: 471758:0:(client.c:2506:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786347111/real 1786347111] req@00000000b149195a x1873120574127104/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786347127 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8720.351202] Lustre: 471760:0:(client.c:2506:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786347114/real 1786347114] req@0000000083619588 x1873120574127232/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786347130 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8721.375542] LustreError: 471757:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000042b02d54 x1873120574128000/t0(0) o250->MGC192.168.203.123@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8721.527668] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8722.400079] Lustre: 471761:0:(client.c:2506:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786347116/real 1786347116] req@00000000946639e4 x1873120574127488/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786347132 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8722.830718] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 8725.782324] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8725.944618] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8726.623083] Lustre: 471758:0:(client.c:2506:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786347121/real 1786347121] req@00000000ad555732 x1873120574127616/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786347137 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8727.213965] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 8729.383183] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8731.036815] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8731.108860] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8741.346330] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8741.346679] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8741.348931] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8741.353703] Lustre: Skipped 1 previous similar message [ 8741.356496] Lustre: Skipped 1 previous similar message [ 8743.890741] Lustre: server umount lustre-OST0000 complete [ 8746.465290] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8746.465955] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8746.469312] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8746.477300] Lustre: Skipped 1 previous similar message [ 8746.485550] Lustre: Skipped 1 previous similar message [ 8751.342912] Lustre: server umount lustre-MDT0000 complete [ 8751.583958] LustreError: 500654:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8751.588212] LustreError: 500654:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 8752.547090] LustreError: 472477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786347163 with bad export cookie 3888953285799155509 [ 8752.549620] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8752.552347] LustreError: 472477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8752.668419] Lustre: server umount lustre-MDT0001 complete [ 8755.960681] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 8756.925698] Key type lgssc unregistered [ 8757.099213] LNet: 501415:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8757.104461] LNetError: 501415:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8757.109253] LNet: Removed LNI 192.168.203.123@tcp [ 8757.112160] LNet: Removed LNI 192.168.203.123@tcp10 [ 8757.114883] LNet: Removed LNI 192.168.203.123@tcp11 [ 8757.117031] LNet: Removed LNI 192.168.203.123@tcp12 [ 8757.119475] LNet: Removed LNI 192.168.203.123@tcp13 [ 8757.122308] LNet: Removed LNI 192.168.203.123@tcp14 [ 8757.125285] LNet: Removed LNI 192.168.203.123@tcp15 [ 8757.127808] LNet: Removed LNI 192.168.203.123@tcp16 [ 8757.129963] LNet: Removed LNI 192.168.203.123@tcp17 [ 8757.132561] LNet: Removed LNI 192.168.203.123@tcp18 [ 8757.134992] LNet: Removed LNI 192.168.203.123@tcp19 [ 8757.137274] LNet: Removed LNI 192.168.203.123@tcp20 [ 8757.139700] LNet: Removed LNI 192.168.203.123@tcp21 [ 8757.142255] LNet: Removed LNI 192.168.203.123@tcp22 [ 8757.144554] LNet: Removed LNI 192.168.203.123@tcp23 [ 8757.146775] LNet: Removed LNI 192.168.203.123@tcp24 [ 8757.149347] LNet: Removed LNI 192.168.203.123@tcp25 [ 8757.151708] LNet: Removed LNI 192.168.203.123@tcp26 [ 8757.153955] LNet: Removed LNI 192.168.203.123@tcp27 [ 8757.156404] LNet: Removed LNI 192.168.203.123@tcp28 [ 8757.158648] LNet: Removed LNI 192.168.203.123@tcp29 [ 8757.160832] LNet: Removed LNI 192.168.203.123@tcp30 [ 8757.162989] LNet: Removed LNI 192.168.203.123@tcp31 [ 8757.165264] LNet: Removed LNI 192.168.203.123@tcp32 [ 8757.167282] LNet: Removed LNI 192.168.203.123@tcp33 [ 8757.169318] LNet: Removed LNI 192.168.203.123@tcp34 [ 8757.171419] LNet: Removed LNI 192.168.203.123@tcp35 [ 8757.173398] LNet: Removed LNI 192.168.203.123@tcp36 [ 8757.175346] LNet: Removed LNI 192.168.203.123@tcp37 [ 8757.177279] LNet: Removed LNI 192.168.203.123@tcp38 [ 8757.179182] LNet: Removed LNI 192.168.203.123@tcp39 [ 8757.181091] LNet: Removed LNI 192.168.203.123@tcp40 [ 8757.185207] LNet: Removed LNI 192.168.203.123@tcp41 [ 8757.187065] LNet: Removed LNI 192.168.203.123@tcp42 [ 8757.188867] LNet: Removed LNI 192.168.203.123@tcp43 [ 8757.190617] LNet: Removed LNI 192.168.203.123@tcp44 [ 8757.192377] LNet: Removed LNI 192.168.203.123@tcp45 [ 8757.194070] LNet: Removed LNI 192.168.203.123@tcp46 [ 8757.195714] LNet: Removed LNI 192.168.203.123@tcp47 [ 8757.197373] LNet: Removed LNI 192.168.203.123@tcp48 [ 8757.198936] LNet: Removed LNI 192.168.203.123@tcp49 [ 8757.206360] LNet: Removed LNI 192.168.203.123@tcp50 [ 8757.586933] Key type .llcrypt unregistered [ 8757.589320] Key type ._llcrypt unregistered [ 8765.643664] Key type ._llcrypt registered [ 8765.644753] Key type .llcrypt registered [ 8765.680410] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 8772.663114] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 8773.056879] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8773.067444] alg: No test for adler32 (adler32-zlib) [ 8773.932851] Lustre: Lustre: Build Version: 2.17.56_50_g8cc021f [ 8774.024743] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 8775.607206] Key type lgssc registered [ 8776.041757] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8786.575846] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8786.582707] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8787.675847] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8787.686941] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8787.723883] Lustre: lustre-MDT0000: new disk, initializing [ 8787.745051] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8787.752659] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8789.120740] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 8793.261144] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8793.327242] Lustre: 504583:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8793.363293] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8793.367218] Lustre: Skipped 1 previous similar message [ 8793.412474] Lustre: lustre-MDT0001: new disk, initializing [ 8793.458933] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8793.485158] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8793.494493] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8794.981184] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 8798.856579] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8800.688755] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8803.206863] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8803.278582] Lustre: *** cfs_fail_loc=90f, val=0*** [ 8803.305907] Lustre: lustre-OST0000: new disk, initializing [ 8803.307496] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8803.309948] Lustre: 506600:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8803.337419] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8805.786064] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 8809.600024] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8811.510403] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8811.513708] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8811.544597] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8816.611540] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8816.612773] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8816.621127] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8816.624830] LustreError: Skipped 1 previous similar message [ 8816.628324] Lustre: Skipped 1 previous similar message [ 8818.272756] Lustre: server umount lustre-OST0000 complete [ 8821.730844] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8821.731927] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8821.740139] Lustre: Skipped 2 previous similar messages [ 8825.695988] Lustre: server umount lustre-MDT0000 complete [ 8827.017115] LustreError: 504577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786347237 with bad export cookie 5022149395695068340 [ 8827.020453] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8827.020506] LustreError: 504577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8827.130871] Lustre: server umount lustre-MDT0001 complete [ 8830.589261] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 8831.485629] Key type lgssc unregistered [ 8831.609382] LNet: 508506:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8831.612896] LNetError: 508506:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8831.625293] LNet: Removed LNI 192.168.203.123@tcp [ 8831.899118] Key type .llcrypt unregistered [ 8831.900753] Key type ._llcrypt unregistered [ 8840.603041] Key type ._llcrypt registered [ 8840.604149] Key type .llcrypt registered [ 8840.642083] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 8840.973898] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8841.037634] alg: No test for adler32 (adler32-zlib) [ 8841.903530] Lustre: Lustre: Build Version: 2.17.56_50_g8cc021f [ 8841.993930] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 8843.567137] Key type lgssc registered [ 8844.034347] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8847.053606] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8847.065600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8848.248478] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8849.890415] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 8853.408091] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8853.554290] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8855.240251] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 8858.194945] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8860.813213] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8864.074656] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8864.259956] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8866.905663] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 8870.336150] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8885.114697] loop0: detected capacity change from 0 to 204800000 [ 8885.169371] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8885.172424] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8885.176601] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8887.026203] loop0: detected capacity change from 0 to 204800000 [ 8887.082564] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8887.087222] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8887.090595] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8889.221200] loop0: detected capacity change from 0 to 204800000 [ 8891.669397] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8891.738107] Lustre: fs15246-OST0000: new disk, initializing [ 8891.741732] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 8891.747280] Lustre: 513370:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8891.782973] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8894.110861] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 8898.385806] loop1: detected capacity change from 0 to 204800000 [ 8900.990510] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8901.063374] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 8901.105179] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 8901.159064] Lustre: fs15246-MDT0000: new disk, initializing [ 8901.225085] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8901.238871] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8901.306515] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 8902.367707] Lustre: 514492:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@0000000062f12ea1 x1873120856148224/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1786347328 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 8903.097881] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 8911.458322] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 8911.462773] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 8911.470726] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 8911.792282] Lustre: server umount fs15246-MDT0000 complete [ 8917.252131] Lustre: server umount fs15246-OST0000 complete [ 8921.571991] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8921.574094] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8921.580629] Lustre: Skipped 1 previous similar message [ 8926.691992] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8926.696333] Lustre: Skipped 2 previous similar messages [ 8927.509753] Lustre: server umount lustre-OST0000 complete [ 8930.272499] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8930.277911] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8930.287420] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8931.810853] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8936.928631] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8936.932535] Lustre: Skipped 3 previous similar messages [ 8942.048126] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8942.051018] Lustre: Skipped 1 previous similar message [ 8943.071113] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8943.136683] Lustre: server umount lustre-MDT0000 complete [ 8944.381891] LustreError: 510493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786347354 with bad export cookie 3053964388437493629 [ 8944.387675] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8944.391921] LustreError: 510493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8944.513808] Lustre: server umount lustre-MDT0001 complete [ 8947.850439] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 8948.839589] Key type lgssc unregistered [ 8948.961280] LNet: 517588:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8948.963850] LNetError: 517588:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8948.973343] LNet: Removed LNI 192.168.203.123@tcp [ 8949.308571] Key type .llcrypt unregistered [ 8949.310178] Key type ._llcrypt unregistered [ 8956.047802] Key type ._llcrypt registered [ 8956.049199] Key type .llcrypt registered [ 8956.086942] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 8956.335794] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8956.352893] alg: No test for adler32 (adler32-zlib) [ 8957.205665] Lustre: Lustre: Build Version: 2.17.56_50_g8cc021f [ 8957.295847] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 8958.871166] Key type lgssc registered [ 8959.222531] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8970.943278] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 8973.789405] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 8986.419293] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8986.425582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8987.505268] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8987.514755] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8987.551057] Lustre: lustre-MDT0000: new disk, initializing [ 8987.571448] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8987.578828] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8989.141668] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 8994.131299] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8994.168771] Lustre: 522037:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8994.191786] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8994.193744] Lustre: Skipped 1 previous similar message [ 8994.228089] Lustre: lustre-MDT0001: new disk, initializing [ 8994.259383] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8994.271313] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8994.275683] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8995.494595] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 8999.350771] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9001.128850] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9003.688302] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9003.793694] Lustre: lustre-OST0000: new disk, initializing [ 9003.795412] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9003.799266] Lustre: 524005:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9003.832067] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9004.547099] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9004.550420] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9004.586864] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9005.790111] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 9009.494864] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9010.916569] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9011.008596] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9012.472876] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9012.556642] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9014.754192] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9014.755438] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9014.757602] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9014.760215] Lustre: Skipped 1 previous similar message [ 9019.550292] Lustre: server umount lustre-OST0000 complete [ 9024.993069] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9024.993209] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9024.994161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9024.994166] Lustre: Skipped 1 previous similar message [ 9024.997103] Lustre: Skipped 1 previous similar message [ 9024.999897] LustreError: Skipped 1 previous similar message [ 9027.187268] Lustre: server umount lustre-MDT0000 complete [ 9028.687274] LustreError: 522830:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786347439 with bad export cookie 11429252953949217572 [ 9028.694059] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9028.695425] LustreError: 522830:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9028.867148] Lustre: server umount lustre-MDT0001 complete [ 9037.200747] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 9040.431308] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 9051.772647] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 8774 sec ========== 03:37:42 (1786347462) [ 9052.281266] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 03:37:42 (1786347462) === [ 9053.340439] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 03:37:43 (1786347463) === [ 9066.177967] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 9067.317652] Key type lgssc unregistered [ 9067.435331] LNet: 530637:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9067.437573] LNetError: 530637:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9067.447324] LNet: Removed LNI 192.168.203.123@tcp [ 9067.721270] Key type .llcrypt unregistered [ 9067.722719] Key type ._llcrypt unregistered