[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 518170727 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001016] APIC: Switch to symmetric I/O mode setup [ 0.003328] x2apic enabled [ 0.004013] Switched APIC routing to physical x2apic. [ 0.006011] kvm-guest: setup PV IPIs [ 0.009000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009034] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010023] pid_max: default: 32768 minimum: 301 [ 0.012180] LSM: Security Framework initializing [ 0.013082] Yama: becoming mindful. [ 0.014063] SELinux: Initializing. [ 0.015129] *** VALIDATE selinux *** [ 0.023634] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028598] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030022] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031145] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032151] *** VALIDATE tmpfs *** [ 0.034154] *** VALIDATE proc *** [ 0.035338] *** VALIDATE cgroup *** [ 0.037012] *** VALIDATE cgroup2 *** [ 0.038338] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039191] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040019] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.042031] Spectre V2 : User space: Vulnerable [ 0.044003] Speculative Store Bypass: Vulnerable [ 0.047895] debug: unmapping init [mem 0xffffffffa8659000-0xffffffffa8660fff] [ 0.050230] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.051833] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.052036] ... version: 2 [ 0.053015] ... bit width: 48 [ 0.054021] ... generic registers: 4 [ 0.055021] ... value mask: 0000ffffffffffff [ 0.056018] ... max period: 00007fffffffffff [ 0.057018] ... fixed-purpose events: 3 [ 0.058017] ... event mask: 000000070000000f [ 0.060307] rcu: Hierarchical SRCU implementation. [ 0.062876] smp: Bringing up secondary CPUs ... [ 0.063765] x86: Booting SMP configuration: [ 0.064060] .... node #0, CPUs: #1 #2 #3 [ 0.067620] smp: Brought up 1 node, 4 CPUs [ 0.069017] smpboot: Max logical packages: 1 [ 0.070028] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.264044] node 0 deferred pages initialised in 192ms [ 0.267013] devtmpfs: initialized [ 0.268307] x86/mm: Memory block size: 128MB [ 0.270902] gcov: version magic: 0x41383552 [ 0.272347] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.273151] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.274426] pinctrl core: initialized pinctrl subsystem [ 0.275312] [ 0.275841] ************************************************************* [ 0.276019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.277018] ** ** [ 0.278020] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.279021] ** ** [ 0.280019] ** This means that this kernel is built to expose internal ** [ 0.281021] ** IOMMU data structures, which may compromise security on ** [ 0.282020] ** your system. ** [ 0.283017] ** ** [ 0.284021] ** If you see this message and you are not debugging the ** [ 0.285023] ** kernel, report this immediately to your vendor! ** [ 0.286020] ** ** [ 0.287018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.288021] ************************************************************* [ 0.289845] NET: Registered protocol family 16 [ 0.290587] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.291090] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.292115] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.293804] cpuidle: using governor menu [ 0.296000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.299643] PCI: Using configuration type 1 for base access [ 0.301177] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.310117] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.311066] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.313078] cryptd: max_cpu_qlen set to 1000 [ 0.315224] ACPI: Added _OSI(Module Device) [ 0.316018] ACPI: Added _OSI(Processor Device) [ 0.317018] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.318012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.322734] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.326016] ACPI: Interpreter enabled [ 0.327099] ACPI: PM: (supports S0 S3 S4 S5) [ 0.328024] ACPI: Using IOAPIC for interrupt routing [ 0.329165] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.330448] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.339970] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.340065] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.341031] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.342099] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.344368] acpiphp: Slot [2] registered [ 0.345214] acpiphp: Slot [5] registered [ 0.346208] acpiphp: Slot [6] registered [ 0.347189] acpiphp: Slot [7] registered [ 0.348195] acpiphp: Slot [8] registered [ 0.349132] acpiphp: Slot [9] registered [ 0.350253] acpiphp: Slot [10] registered [ 0.351225] acpiphp: Slot [3] registered [ 0.352151] acpiphp: Slot [4] registered [ 0.353145] acpiphp: Slot [11] registered [ 0.354140] acpiphp: Slot [12] registered [ 0.355130] acpiphp: Slot [13] registered [ 0.356154] acpiphp: Slot [14] registered [ 0.357176] acpiphp: Slot [15] registered [ 0.358188] acpiphp: Slot [16] registered [ 0.359142] acpiphp: Slot [17] registered [ 0.360173] acpiphp: Slot [18] registered [ 0.361154] acpiphp: Slot [19] registered [ 0.362137] acpiphp: Slot [20] registered [ 0.363146] acpiphp: Slot [21] registered [ 0.364188] acpiphp: Slot [22] registered [ 0.365147] acpiphp: Slot [23] registered [ 0.366137] acpiphp: Slot [24] registered [ 0.367135] acpiphp: Slot [25] registered [ 0.368234] acpiphp: Slot [26] registered [ 0.369171] acpiphp: Slot [27] registered [ 0.370163] acpiphp: Slot [28] registered [ 0.371176] acpiphp: Slot [29] registered [ 0.372144] acpiphp: Slot [30] registered [ 0.373162] acpiphp: Slot [31] registered [ 0.374101] PCI host bridge to bus 0000:00 [ 0.375037] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.376037] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.377039] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.378041] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.379041] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.380043] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.381290] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.383152] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.385439] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.391022] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.395072] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.396033] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.397049] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.398047] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.399747] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.400879] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.401071] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.402777] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.404931] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.412025] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.415024] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.419309] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.422027] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.425025] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.432045] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.438085] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.442026] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.445024] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.453030] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.460142] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.463022] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.467021] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.475031] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.483055] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.486026] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.489029] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.496032] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.502958] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.505029] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.508026] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.516032] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.524402] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.537022] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.553027] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.584025] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.604780] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.607255] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.609251] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.610329] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.611280] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.615058] iommu: Default domain type: Passthrough [ 0.617455] SCSI subsystem initialized [ 0.618131] ACPI: bus type USB registered [ 0.620100] usbcore: registered new interface driver usbfs [ 0.621062] usbcore: registered new interface driver hub [ 0.622128] usbcore: registered new device driver usb [ 0.624145] pps_core: LinuxPPS API ver. 1 registered [ 0.625012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.628072] PTP clock support registered [ 0.630132] EDAC MC: Ver: 3.0.0 [ 0.632044] PCI: Using ACPI for IRQ routing [ 0.634052] NetLabel: Initializing [ 0.635020] NetLabel: domain hash size = 128 [ 0.637017] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.640168] NetLabel: unlabeled traffic allowed by default [ 0.642212] vgaarb: loaded [ 0.644221] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.646035] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.653000] clocksource: Switched to clocksource kvm-clock [ 0.762608] VFS: Disk quotas dquot_6.6.0 [ 0.764944] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.767931] *** VALIDATE ramfs *** [ 0.769551] *** VALIDATE hugetlbfs *** [ 0.773209] pnp: PnP ACPI init [ 0.775990] pnp: PnP ACPI: found 6 devices [ 0.795540] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.799729] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.801953] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.803517] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.805690] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.807716] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.810706] NET: Registered protocol family 2 [ 0.813403] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.818510] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.820693] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.825729] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.828894] TCP: Hash tables configured (established 65536 bind 65536) [ 0.832089] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.834434] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.836878] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.839331] NET: Registered protocol family 1 [ 0.841120] RPC: Registered named UNIX socket transport module. [ 0.842300] RPC: Registered udp transport module. [ 0.843353] RPC: Registered tcp transport module. [ 0.844395] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.845585] NET: Registered protocol family 44 [ 0.846524] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.848096] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.849261] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.850971] PCI: CLS 0 bytes, default 64 [ 0.851951] Unpacking initramfs... [ 2.301848] debug: unmapping init [mem 0xffff9d7cbcc54000-0xffff9d7cbffbffff] [ 2.306101] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.308217] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.310722] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.865711] Initialise system trusted keyrings [ 2.867149] Key type blacklist registered [ 2.868848] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.876546] zbud: loaded [ 2.879356] *** VALIDATE nfs *** [ 2.880459] *** VALIDATE nfs4 *** [ 2.882061] pstore: using deflate compression [ 2.885419] Platform Keyring initialized [ 2.999299] NET: Registered protocol family 38 [ 3.000879] Key type asymmetric registered [ 3.001922] Asymmetric key parser 'x509' registered [ 3.003460] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.006683] io scheduler mq-deadline registered [ 3.008730] io scheduler kyber registered [ 3.011283] io scheduler bfq registered [ 3.013301] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.015551] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.017832] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.019620] ACPI: Power Button [PWRF] [ 3.023377] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.028268] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.037629] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.042320] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.051607] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.077430] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.106306] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.111501] Non-volatile memory driver v1.3 [ 3.113296] Linux agpgart interface v0.103 [ 3.148480] virtio_blk virtio1: [vda] 145904 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.151731] vda: detected capacity change from 0 to 74702848 [ 3.171323] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.174883] vdb: detected capacity change from 0 to 1073741824 [ 3.193250] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.196164] vdc: detected capacity change from 0 to 2621440000 [ 3.210758] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.213645] vdd: detected capacity change from 0 to 2621440000 [ 3.227741] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.231143] vde: detected capacity change from 0 to 4294967296 [ 3.247136] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.249539] vdf: detected capacity change from 0 to 4294967296 [ 3.257671] libphy: Fixed MDIO Bus: probed [ 3.262698] usbcore: registered new interface driver usbserial_generic [ 3.264855] usbserial: USB Serial support registered for generic [ 3.267445] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.271936] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.273802] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.276179] mousedev: PS/2 mouse device common for all mice [ 3.278328] rtc_cmos 00:05: RTC can wake from S4 [ 3.280703] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.282361] rtc_cmos 00:05: registered as rtc0 [ 3.285860] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.286290] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.290453] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.292082] intel_pstate: CPU model not supported [ 3.297362] hid: raw HID events driver (C) Jiri Kosina [ 3.299824] usbcore: registered new interface driver usbhid [ 3.302161] usbhid: USB HID core driver [ 3.303684] drop_monitor: Initializing network drop monitor service [ 3.306253] Initializing XFRM netlink socket [ 3.308446] NET: Registered protocol family 10 [ 3.312112] Segment Routing with IPv6 [ 3.313589] NET: Registered protocol family 17 [ 3.315809] mpls_gso: MPLS GSO support [ 3.321997] RAS: Correctable Errors collector initialized. [ 3.323941] AVX version of gcm_enc/dec engaged. [ 3.325412] AES CTR mode by8 optimization enabled [ 3.431867] sched_clock: Marking stable (3431847801, 0)->(4701813677, -1269965876) [ 3.434358] registered taskstats version 1 [ 3.435809] Loading compiled-in X.509 certificates [ 3.437983] zswap: loaded using pool lzo/zbud [ 3.459100] Key type big_key registered [ 3.470931] Key type encrypted registered [ 3.472507] ima: No TPM chip found, activating TPM-bypass! [ 3.474231] ima: Allocated hash algorithm: sha1 [ 3.475607] ima: No architecture policies found [ 3.476879] evm: Initialising EVM extended attributes: [ 3.478596] evm: security.selinux [ 3.479907] evm: security.ima [ 3.481011] evm: security.capability [ 3.482240] evm: HMAC attrs: 0x1 [ 3.485482] rtc_cmos 00:05: setting system clock to 2026-08-13 13:34:45 UTC (1786628085) [ 3.491681] debug: unmapping init [mem 0xffffffffa9603000-0xffffffffa97fffff] [ 3.494516] debug: unmapping init [mem 0xffffffffa8382000-0xffffffffa8658fff] [ 3.503084] Write protecting the kernel read-only data: 28672k [ 3.506473] debug: unmapping init [mem 0xffffffffa6a03000-0xffffffffa6bfffff] [ 3.509435] debug: unmapping init [mem 0xffffffffa7314000-0xffffffffa73fffff] [ 3.553748] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.564259] systemd[1]: Detected virtualization kvm. [ 3.566292] systemd[1]: Detected architecture x86-64. [ 3.568303] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.593251] systemd[1]: No hostname configured. [ 3.595027] systemd[1]: Set hostname to . [ 3.597168] random: systemd: uninitialized urandom read (16 bytes read) [ 3.599795] systemd[1]: Initializing machine ID from random generator. [ 3.644826] random: ln: uninitialized urandom read (6 bytes read) [ 3.797211] random: systemd: uninitialized urandom read (16 bytes read) [ 3.800524] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.806209] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.813984] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Swap. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... [ OK ] Reached target Sockets. [ OK ] Reached target Timers. Starting Journal Service... [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.498869] device-mapper: uevent: version 1.0.3 [ 4.500603] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.360244] random: fast init done [ 5.362648] virtio_net virtio0 ens2: renamed from eth0 [ 5.369432] scsi host0: ata_piix [ 5.393224] scsi host1: ata_piix [ 5.397499] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.400557] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.369252] dracut-initqueue[586]: RTNETLINK answers: File exists [ 9.988618] random: crng init done [ 9.990149] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.614476] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.884777] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.269441] SELinux: Disabled at runtime. [ 12.334672] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.343380] systemd[1]: Detected virtualization kvm. [ 12.345451] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.022942] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.026138] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.030953] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.033763] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.036083] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.045870] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.055506] systemd[1]: Listening on RPCbind Server Activation Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Stopped target Switch Root. [ OK ] Reached target rpc_pipefs.target. Starting Remount Root and Kernel File Systems... Mounting POSIX Message Queue File System... [ OK ] Listening on Process Core Dump Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting Kernel Debug File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Activating swap /dev/disk/by-label/SWAP... Mounting Huge Pages File System... [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Kernel Socket. Starting Apply Kernel Variables... [ OK ] Stopped target Initrd File Systems. [[ 13.224985] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS  OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-getty.slice. [ OK ] Reached target Slices. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.569243] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.837669] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.896677] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.997315] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.017584] EDAC sbridge: Ver: 1.1.2 [ 15.286637] Key type dns_resolver registered [ 15.582309] NFS: Registering the id_resolver key type [ 15.584498] Key type id_resolver registered [ 15.586101] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg232-server login: [ 43.152972] libcfs: loading out-of-tree module taints kernel. [ 43.173252] Key type ._llcrypt registered [ 43.174622] Key type .llcrypt registered [ 43.218268] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [ 62.636010] hrtimer: interrupt took 3611876 ns [ 62.908055] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 64.908230] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 64.915983] alg: No test for adler32 (adler32-zlib) [ 66.512396] Lustre: Lustre: Build Version: 2.17.57_2_g7a6d6aa [ 67.538603] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 69.287358] Key type lgssc registered [ 71.142875] Lustre: Echo OBD driver; http://www.lustre.org/ [ 92.030431] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 140.982839] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 157.457931] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 157.509942] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 158.861614] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 158.893698] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 159.051152] Lustre: lustre-MDT0000: new disk, initializing [ 159.154897] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 159.178251] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 163.989552] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 178.415642] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 178.522247] Lustre: 6511:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 178.559535] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 178.564546] Lustre: Skipped 1 previous similar message [ 178.704336] Lustre: lustre-MDT0001: new disk, initializing [ 178.841802] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 178.873613] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 178.886425] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 183.885703] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 189.506528] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 199.844203] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 200.348277] Lustre: lustre-OST0000: new disk, initializing [ 200.352872] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 200.360883] Lustre: 8449:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 200.443339] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 204.449319] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 204.454315] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 204.583994] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 208.171545] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 223.255109] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 223.406339] Lustre: lustre-OST0001: new disk, initializing [ 223.409594] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 223.418786] Lustre: 9523:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 223.474573] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 229.266775] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 232.474633] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 232.486954] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 232.608772] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 242.638995] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 250.849258] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 258.597639] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing check_logdir /tmp/testlogs/ [ 264.533883] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing yml_node [ 271.816104] Lustre: DEBUG MARKER: Client: 2.17.57.2 [ 275.025651] Lustre: DEBUG MARKER: MDS: 2.17.57.2 [ 278.313504] Lustre: DEBUG MARKER: OSS: 2.17.57.2 [ 280.273208] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Aug 13 09:39:19 EDT 2026 [ 301.496726] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 303.094657] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 322.530951] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 322.550352] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 322.575864] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 324.585270] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 324.590207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 324.619085] Lustre: Skipped 1 previous similar message [ 324.637742] Lustre: Skipped 3 previous similar messages [ 329.713023] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 334.303114] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 334.552283] Lustre: server umount lustre-MDT0000 complete [ 338.557319] LustreError: 6503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786628420 with bad export cookie 18207068210817965796 [ 338.571860] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 339.935817] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 339.935826] LustreError: 8361:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 339.936282] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 339.936286] Lustre: Skipped 3 previous similar messages [ 339.954472] Lustre: Skipped 2 previous similar messages [ 339.997282] LustreError: 8361:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 343.971852] LustreError: 12467:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 343.973097] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 343.999314] LustreError: 12467:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 344.025000] Lustre: Skipped 2 previous similar messages [ 345.199973] Lustre: server umount lustre-MDT0001 complete [ 350.009429] Lustre: server umount lustre-OST0000 complete [ 354.283861] Lustre: server umount lustre-OST0001 complete [ 361.307991] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [ 371.386100] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 415.492530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 415.859405] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 415.904386] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 416.012584] Lustre: lustre-MDT0000: new disk, initializing [ 416.119230] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 416.137064] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 421.420738] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 433.834203] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 434.039283] Lustre: 15082:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 434.084723] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 434.087911] Lustre: Skipped 1 previous similar message [ 434.187571] Lustre: lustre-MDT0001: new disk, initializing [ 434.379899] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 434.442280] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 434.455230] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 439.785704] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 449.414905] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 456.078952] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 463.148643] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 463.432670] Lustre: lustre-OST0000: new disk, initializing [ 463.436035] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 463.448429] Lustre: 17050:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 463.550412] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 465.538046] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 465.554848] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 465.754722] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 471.059900] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 480.730564] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 485.506636] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 485.706135] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 490.395268] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 490.708745] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 496.617207] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 496.618215] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 496.629463] LustreError: Skipped 1 previous similar message [ 496.631454] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 496.682159] Lustre: Skipped 1 previous similar message [ 507.359351] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 507.539727] Lustre: server umount lustre-OST0000 complete [ 511.456063] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 511.465983] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 517.089549] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 517.098672] Lustre: Skipped 7 previous similar messages [ 517.765327] Lustre: server umount lustre-MDT0000 complete [ 521.145131] LustreError: 18389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786628603 with bad export cookie 18207068210817967329 [ 521.146613] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 521.153995] LustreError: 18389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 521.477277] Lustre: server umount lustre-MDT0001 complete [ 526.489329] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 09:43:25 (1786628605) [ 529.309970] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 531.452540] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 09:43:30 (1786628610) [ 537.905107] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 538.504318] LustreError: 19462:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 538.576628] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 543.124559] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 543.716622] LustreError: 19463:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 548.843542] LustreError: 19462:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 550.861787] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 555.561802] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 564.197752] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 571.314833] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 579.972748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 585.081645] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 594.333407] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 607.203846] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 607.208330] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 607.223650] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 607.233787] Lustre: Skipped 2 previous similar messages [ 607.246933] Lustre: Skipped 1 previous similar message [ 612.680469] Lustre: server umount lustre-OST0000 complete [ 622.852531] Lustre: server umount lustre-MDT0000 complete [ 627.051976] LustreError: 19442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786628709 with bad export cookie 18207068210817968211 [ 627.068237] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 627.074899] LustreError: 19442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 627.588743] Lustre: server umount lustre-MDT0001 complete [ 639.417858] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 642.453693] Key type lgssc unregistered [ 642.847181] LNet: 22997:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 642.853992] LNetError: 22997:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 642.868342] LNet: Removed LNI 192.168.202.132@tcp [ 643.884922] Key type .llcrypt unregistered [ 643.886778] Key type ._llcrypt unregistered [ 671.421792] Key type ._llcrypt registered [ 671.427156] Key type .llcrypt registered [ 671.616860] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 673.024498] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 673.157423] alg: No test for adler32 (adler32-zlib) [ 674.622316] Lustre: Lustre: Build Version: 2.17.57_2_g7a6d6aa [ 675.105690] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 676.888178] Key type lgssc registered [ 678.303506] Lustre: Echo OBD driver; http://www.lustre.org/ [ 688.249903] loop: module loaded [ 688.434395] loop0: detected capacity change from 0 to 204800000 [ 688.839992] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 688.851064] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 689.229662] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 695.768464] loop0: detected capacity change from 0 to 204800000 [ 696.159967] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 696.169150] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 696.180715] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 702.299528] loop0: detected capacity change from 0 to 204800000 [ 709.012729] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 709.042379] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 710.367220] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 710.459309] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 710.595415] Lustre: lustre-MDT0000: new disk, initializing [ 710.720760] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 710.747649] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 716.160446] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 725.670852] loop1: detected capacity change from 0 to 204800000 [ 732.110782] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 732.474667] Lustre: lustre-OST07c6: new disk, initializing [ 732.493518] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 732.513268] Lustre: Skipped 1 previous similar message [ 732.528351] Lustre: 26188:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 732.651342] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 739.454827] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 741.013268] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 741.027099] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 741.063709] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 744.254296] Lustre: 25172:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 744.268111] Lustre: 25172:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 744.279747] Lustre: 25172:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 744.287899] Lustre: 25172:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 744.298503] Lustre: 25172:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 744.303540] Lustre: 25172:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 760.494845] Lustre: 26805:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 760.511268] Lustre: 26805:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 760.520689] Lustre: 26805:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 760.528572] Lustre: 26805:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 760.541575] Lustre: 26805:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 760.551263] Lustre: 26805:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 765.741915] Lustre: 25173:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 765.752431] Lustre: 25173:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 765.763594] Lustre: 25173:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 765.770729] Lustre: 25173:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 765.773907] Lustre: 25173:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 765.781377] Lustre: 25173:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 765.790214] Lustre: 25173:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 765.798419] Lustre: 25173:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 765.802455] Lustre: 25173:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 765.811553] Lustre: 25173:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 765.822346] Lustre: 25173:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 765.829688] Lustre: 25173:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 771.014958] Lustre: 26805:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 257 < left 278, rollback = 2 [ 771.028206] Lustre: 26805:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 771.043147] Lustre: 26805:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 771.053026] Lustre: 26805:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 771.060033] Lustre: 26805:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 771.071868] Lustre: 26805:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 771.081844] Lustre: 26805:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 771.094588] Lustre: 26805:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 771.106023] Lustre: 26805:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 771.118021] Lustre: 26805:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 771.128376] Lustre: 26805:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 771.135723] Lustre: 26805:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 776.086985] Lustre: 25173:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 776.100206] Lustre: 25173:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 776.107878] Lustre: 25173:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 776.115325] Lustre: 25173:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 776.128984] Lustre: 25173:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 776.139968] Lustre: 25173:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 776.150077] Lustre: 25173:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 776.160055] Lustre: 25173:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 776.171814] Lustre: 25173:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 776.198415] Lustre: 25173:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 776.215241] Lustre: 25173:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 776.233680] Lustre: 25173:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 784.524588] Lustre: 26805:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 784.542746] Lustre: 26805:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 784.551819] Lustre: 26805:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 784.560899] Lustre: 26805:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 784.572852] Lustre: 26805:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 784.579288] Lustre: 26805:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 784.590732] Lustre: 26805:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 784.599872] Lustre: 26805:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 784.606928] Lustre: 26805:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 784.618580] Lustre: 26805:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 784.627636] Lustre: 26805:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 784.638795] Lustre: 26805:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 801.220871] Lustre: 25172:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 801.236922] Lustre: 25172:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 7 previous similar messages [ 801.243121] Lustre: 25172:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 801.249033] Lustre: 25172:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 801.253995] Lustre: 25172:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 801.259149] Lustre: 25172:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 801.264060] Lustre: 25172:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 801.275649] Lustre: 25172:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 801.284893] Lustre: 25172:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 801.295474] Lustre: 25172:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 801.306675] Lustre: 25172:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 801.316958] Lustre: 25172:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 838.114262] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 838.129187] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 842.917931] Lustre: server umount lustre-OST07c6 complete [ 853.486343] Lustre: server umount lustre-MDT0000 complete [ 872.229124] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 874.972780] Key type lgssc unregistered [ 875.276858] LNet: 28750:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 875.299013] LNetError: 28750:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 875.319315] LNet: Removed LNI 192.168.202.132@tcp [ 876.277173] Key type .llcrypt unregistered [ 876.280135] Key type ._llcrypt unregistered [ 897.023253] Key type ._llcrypt registered [ 897.036441] Key type .llcrypt registered [ 897.123563] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 898.958588] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 898.984322] alg: No test for adler32 (adler32-zlib) [ 900.119862] Lustre: Lustre: Build Version: 2.17.57_2_g7a6d6aa [ 900.396534] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 902.128223] Key type lgssc registered [ 903.527780] Lustre: Echo OBD driver; http://www.lustre.org/ [ 911.732617] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 911.745864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 913.304672] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 917.986792] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 923.619790] LustreError: 30022:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 923.638899] LustreError: 30022:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 926.853845] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 927.322659] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 932.465555] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 940.741584] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 946.167486] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 954.239087] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 954.696671] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 958.813710] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 961.845168] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 971.218392] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 989.666758] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 989.676690] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 989.694988] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 992.750781] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 992.754520] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 992.774076] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 994.645147] Lustre: server umount lustre-OST0000 complete [ 998.889665] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 998.895719] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 998.904576] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 998.917262] Lustre: Skipped 1 previous similar message [ 1002.981783] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1002.983505] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1005.016617] Lustre: server umount lustre-MDT0000 complete [ 1008.100541] LustreError: 30018:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1008.135418] LustreError: 30018:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1009.180063] LustreError: 30756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786629091 with bad export cookie 11607592337089989365 [ 1009.194466] LustreError: 30756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1009.195816] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1009.651759] Lustre: server umount lustre-MDT0001 complete [ 1021.012194] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 1024.077590] Key type lgssc unregistered [ 1024.415645] LNet: 33759:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1024.429925] LNetError: 33759:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1024.459338] LNet: Removed LNI 192.168.202.132@tcp [ 1025.481836] Key type .llcrypt unregistered [ 1025.484869] Key type ._llcrypt unregistered [ 1048.290786] Key type ._llcrypt registered [ 1048.292479] Key type .llcrypt registered [ 1048.386196] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 1049.467934] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1049.584640] alg: No test for adler32 (adler32-zlib) [ 1050.759367] Lustre: Lustre: Build Version: 2.17.57_2_g7a6d6aa [ 1051.155678] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 1052.867718] Key type lgssc registered [ 1054.428220] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1065.183400] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1065.212116] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1066.867082] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1072.024103] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1076.202737] LustreError: 35034:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1076.228189] LustreError: 35034:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1081.199759] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1081.319648] LustreError: 35033:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1081.724649] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1087.040671] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1094.847322] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1101.712513] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1109.814961] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1110.322623] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1111.359113] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1118.615936] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1127.914339] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1142.243575] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1142.247703] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1142.252413] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1142.275643] Lustre: Skipped 1 previous similar message [ 1147.311516] Lustre: server umount lustre-OST0000 complete [ 1152.482790] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1152.494840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1152.505432] Lustre: Skipped 1 previous similar message [ 1152.510524] Lustre: Skipped 1 previous similar message [ 1157.603945] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1157.616822] Lustre: Skipped 2 previous similar messages [ 1157.813917] Lustre: server umount lustre-MDT0000 complete [ 1162.726310] LustreError: 35033:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1162.756989] LustreError: 35033:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1163.109444] LustreError: 35014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786629245 with bad export cookie 1243045673558139886 [ 1163.127838] LustreError: 35014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1163.132782] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1163.928756] Lustre: server umount lustre-MDT0001 complete [ 1177.548686] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 1181.509371] Key type lgssc unregistered [ 1181.894864] LNet: 38595:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1181.898764] LNetError: 38595:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1181.920120] LNet: Removed LNI 192.168.202.132@tcp [ 1183.298048] Key type .llcrypt unregistered [ 1183.301264] Key type ._llcrypt unregistered [ 1206.505326] Key type ._llcrypt registered [ 1206.510132] Key type .llcrypt registered [ 1206.617385] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 1208.300646] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1208.386870] alg: No test for adler32 (adler32-zlib) [ 1209.534985] Lustre: Lustre: Build Version: 2.17.57_2_g7a6d6aa [ 1209.907479] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 1211.695194] Key type lgssc registered [ 1213.706813] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1223.337613] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1223.365810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1224.988222] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1229.943908] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1234.407567] LustreError: 40009:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1234.421522] LustreError: 40009:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1239.521035] LustreError: 40008:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1239.735155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1240.085351] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1246.154715] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1255.573685] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1262.462488] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1270.192739] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1270.704514] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1278.572156] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1287.580615] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1294.334924] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1300.826590] Lustre: server umount lustre-MDT0000 complete [ 1301.499140] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1301.510853] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1301.534518] LustreError: 40009:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1304.550835] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1304.551489] LustreError: 42306:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1304.565112] LustreError: 42306:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1305.073343] LustreError: 39988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786629387 with bad export cookie 13453131177106712471 [ 1305.078858] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1305.086799] LustreError: 39988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1307.744814] Lustre: server umount lustre-MDT0001 complete [ 1320.950090] LustreError: 39989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.32@tcp arrived at 1786629402 with bad export cookie 13453131177106713136 [ 1320.965425] LustreError: 39989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1340.600568] Lustre: server umount lustre-OST0000 complete [ 1359.591498] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 1362.689886] Key type lgssc unregistered [ 1363.036029] LNet: 43898:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1363.042420] LNetError: 43898:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1363.065385] LNet: Removed LNI 192.168.202.132@tcp [ 1363.933785] Key type .llcrypt unregistered [ 1363.942465] Key type ._llcrypt unregistered [ 1390.576300] Key type ._llcrypt registered [ 1390.578257] Key type .llcrypt registered [ 1390.703944] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 1392.010372] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1392.196085] alg: No test for adler32 (adler32-zlib) [ 1393.405927] Lustre: Lustre: Build Version: 2.17.57_2_g7a6d6aa [ 1393.735247] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 1395.447190] Key type lgssc registered [ 1396.938468] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1406.439663] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1406.466972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1407.893030] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1413.399687] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1417.187528] LustreError: 45318:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1417.206306] LustreError: 45318:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1422.308433] LustreError: 45317:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1422.443313] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1423.002450] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1427.988715] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1437.316943] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1444.397213] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1453.022536] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1453.502767] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1461.541213] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1471.003486] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1477.051090] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1483.235339] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1483.253499] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1483.270246] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1487.333331] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1487.334731] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1487.335827] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1487.338959] Lustre: Skipped 2 previous similar messages [ 1488.373959] Lustre: lustre-OST0000: Not available for connect from 192.168.202.32@tcp (stopping) [ 1492.462541] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1492.476441] Lustre: Skipped 1 previous similar message [ 1497.055290] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1497.333238] Lustre: server umount lustre-OST0000 complete [ 1518.049678] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1518.078793] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1518.090141] Lustre: Skipped 1 previous similar message [ 1518.122321] Lustre: Skipped 2 previous similar messages [ 1521.948160] Lustre: server umount lustre-MDT0000 complete [ 1522.213846] LustreError: 47088:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1522.258638] LustreError: 47088:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1527.280376] LustreError: 45312:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1527.322957] LustreError: 45312:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1527.447241] LustreError: 45299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786629609 with bad export cookie 6647322946845135199 [ 1527.466234] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1527.468964] LustreError: 45299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1534.403868] Lustre: server umount lustre-MDT0001 complete [ 1548.281115] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 1551.058498] Key type lgssc unregistered [ 1551.394937] LNet: 49033:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1551.402439] LNetError: 49033:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1551.416707] LNet: Removed LNI 192.168.202.132@tcp [ 1552.478765] Key type .llcrypt unregistered [ 1552.481256] Key type ._llcrypt unregistered [ 1576.866451] Key type ._llcrypt registered [ 1576.869535] Key type .llcrypt registered [ 1576.976327] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 1577.891513] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1577.922577] alg: No test for adler32 (adler32-zlib) [ 1579.183158] Lustre: Lustre: Build Version: 2.17.57_2_g7a6d6aa [ 1579.455944] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 1581.207250] Key type lgssc registered [ 1582.814976] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1592.028928] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1592.056524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1593.670737] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1599.328206] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1603.042237] LustreError: 50462:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1603.059579] LustreError: 50462:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1608.163910] LustreError: 50463:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1609.543456] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1610.020528] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1615.771155] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1623.953818] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1630.295842] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1638.058405] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1638.408548] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1639.422292] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1645.599853] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1654.132337] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1660.837545] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1665.054357] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1677.425029] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-1310h02m36s [ 1678.937730] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1681.895960] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1681.901083] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1681.913379] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.32@tcp (stopping) [ 1681.937319] Lustre: Skipped 2 previous similar messages [ 1682.976691] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1682.978554] Lustre: Skipped 1 previous similar message [ 1686.969302] Lustre: server umount lustre-MDT0000 complete [ 1687.012378] LustreError: 50458:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1689.058792] LustreError: 50457:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1689.084335] LustreError: 50457:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1690.758259] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1693.165845] LustreError: 52956:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1699.126312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1699.203773] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1699.302399] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.32@tcp (not set up) [ 1699.386811] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1699.434202] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1700.440947] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1703.885982] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1708.756945] Lustre: DEBUG MARKER: Wait for df (36784) ... [ 1710.631026] Lustre: DEBUG MARKER: done [ 1719.786419] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1719.800580] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1719.824153] Lustre: Skipped 2 previous similar messages [ 1719.836180] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1720.834893] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1723.183064] Lustre: server umount lustre-OST0000 complete [ 1731.041551] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1731.073458] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1731.080023] Lustre: Skipped 3 previous similar messages [ 1732.891547] Lustre: server umount lustre-MDT0000 complete [ 1737.018139] LustreError: 51197:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786629819 with bad export cookie 18412676152453016360 [ 1737.035276] LustreError: 51197:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1737.040738] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1737.604924] Lustre: server umount lustre-MDT0001 complete [ 1749.294056] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 1752.210543] Key type lgssc unregistered [ 1752.515274] LNet: 55391:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1752.522411] LNetError: 55391:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1752.540824] LNet: Removed LNI 192.168.202.132@tcp [ 1753.626328] Key type .llcrypt unregistered [ 1753.629698] Key type ._llcrypt unregistered [ 1797.050424] Key type ._llcrypt registered [ 1797.052949] Key type .llcrypt registered [ 1797.191305] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 1798.703449] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1798.747935] alg: No test for adler32 (adler32-zlib) [ 1799.936075] Lustre: Lustre: Build Version: 2.17.57_2_g7a6d6aa [ 1800.335536] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 1802.026967] Key type lgssc registered [ 1803.584504] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1811.923815] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1811.936287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1813.266634] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1813.292291] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1813.468788] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1818.240226] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1827.416995] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1827.519136] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1827.546702] Lustre: 57603:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1827.762299] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1832.951099] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1841.383978] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1847.003742] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1854.088585] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1854.230709] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1854.482288] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1861.795806] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1861.798721] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1869.593426] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1876.254787] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-1310h05m55s [ 1877.650193] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1885.510915] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1887.718157] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1887.867859] LustreError: 59058:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1908.243188] Lustre: lustre-MDT0000: Client 3f535a92-5e11-44d4-8568-50c6fe1fe144 (at 192.168.202.32@tcp) reconnecting [ 1927.943886] LustreError: 59058:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1930.272912] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1932.906879] Lustre: DEBUG MARKER: done [ 1941.658864] Lustre: server umount lustre-OST0000 complete [ 1945.568507] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1945.572432] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1945.588919] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1946.668159] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1946.682189] Lustre: Skipped 2 previous similar messages [ 1952.739143] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1958.884418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1958.911370] Lustre: Skipped 2 previous similar messages [ 1959.391174] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1959.719518] Lustre: server umount lustre-MDT0000 complete [ 1960.937089] Lustre: 56925:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786630026/real 1786630026] req@00000000954b875e x1873417215205120/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786630042 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1960.992269] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1961.012262] Lustre: Skipped 1 previous similar message [ 1961.952573] Lustre: 56923:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786630027/real 1786630027] req@000000007624e0a5 x1873417215207424/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786630043 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1962.975217] Lustre: 56926:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786630028/real 1786630028] req@00000000ba1c0b9f x1873417215208192/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786630044 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1965.643684] LustreError: 57593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786630047 with bad export cookie 7846786759830372156 [ 1965.666635] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1965.669251] LustreError: 57593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1967.049706] Lustre: server umount lustre-MDT0001 complete [ 1983.682930] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 1987.053116] Key type lgssc unregistered [ 1987.466521] LNet: 61940:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1987.506201] LNetError: 61940:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1988.530044] LNet: Removed LNI 192.168.202.132@tcp [ 1989.586909] Key type .llcrypt unregistered [ 1989.591291] Key type ._llcrypt unregistered [ 2034.771401] Key type ._llcrypt registered [ 2034.773918] Key type .llcrypt registered [ 2034.885854] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 2036.103558] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2036.182317] alg: No test for adler32 (adler32-zlib) [ 2037.240436] Lustre: Lustre: Build Version: 2.17.57_2_g7a6d6aa [ 2037.506915] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 2039.279143] Key type lgssc registered [ 2040.844351] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2045.062878] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2045.078574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2046.332885] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2046.360908] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2046.573556] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2047.785275] Lustre: Failing over lustre-MDT0000 [ 2048.215293] Lustre: server umount lustre-MDT0000 complete [ 2055.227375] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 10:08:54 (1786630134) [ 2062.520594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2062.881088] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2067.813955] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2077.762406] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2077.924755] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2077.974121] Lustre: 64744:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2078.368626] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2083.315254] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2091.576692] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2098.070836] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2104.788410] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2104.939067] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2105.109812] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2111.040741] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2111.152978] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2118.487807] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2124.604306] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2135.010864] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2135.024811] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2135.049135] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2137.064419] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2137.066744] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2137.089211] Lustre: Skipped 2 previous similar messages [ 2138.954064] Lustre: server umount lustre-MDT0000 complete [ 2143.077141] LustreError: 64735:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786630225 with bad export cookie 9238187363270966746 [ 2143.084936] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2143.091548] LustreError: 64735:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2143.522120] Lustre: server umount lustre-MDT0001 complete [ 2147.270669] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2152.321446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2163.535173] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2163.683750] Lustre: 63666:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786630229/real 1786630229] req@000000000314a9b6 x1873417463702272/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786630245 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2163.715740] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2163.725745] Lustre: Skipped 1 previous similar message [ 2164.705943] Lustre: 63665:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786630230/real 1786630230] req@00000000e92c58e7 x1873417463702400/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786630246 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2168.799438] Lustre: 63664:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786630234/real 1786630234] req@00000000d679ee39 x1873417463702528/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786630250 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2168.864139] LustreError: 63662:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000091f3e4db x1873417463703424/t0(0) o250->MGC192.168.202.132@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2169.207780] LustreError: 66594:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2169.227285] LustreError: 66594:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2169.326730] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2169.409559] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2172.330617] LustreError: 66595:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2173.347502] LustreError: 68515:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2173.370241] LustreError: 68515:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2174.367154] Lustre: 63665:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786630240/real 1786630240] req@00000000d9e0eb47 x1873417463702912/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786630256 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2174.405738] Lustre: 63665:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2174.709387] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2178.533928] LustreError: 68515:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2178.558596] LustreError: 68515:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2179.551088] Lustre: 63665:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786630245/real 1786630245] req@00000000eb4216c7 x1873417463703040/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786630261 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2183.652781] LustreError: 68516:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2183.685177] LustreError: 68516:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2185.005400] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2185.407732] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2190.625329] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2190.840237] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2192.964089] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2201.288998] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2208.798623] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2218.466867] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2218.475539] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2218.494841] Lustre: Skipped 1 previous similar message [ 2223.585466] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2223.589689] Lustre: Skipped 2 previous similar messages [ 2224.393720] Lustre: server umount lustre-MDT0000 complete [ 2228.670539] LustreError: 65479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786630310 with bad export cookie 9238187363270969028 [ 2228.676854] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2228.686063] LustreError: 65479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2228.723328] LustreError: 68516:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2228.740184] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2228.774811] Lustre: Skipped 1 previous similar message [ 2228.792783] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2229.105982] Lustre: server umount lustre-MDT0001 complete [ 2236.215200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2246.275366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2253.660469] LustreError: 66594:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2253.729614] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2253.784789] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2258.201046] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2258.856571] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2266.522204] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2268.083158] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2271.448955] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2279.222493] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2284.567901] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2292.615176] Lustre: server umount lustre-MDT0000 complete [ 2292.707427] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2292.716574] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2292.736263] LustreError: 69251:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2292.755875] LustreError: 69251:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2296.531654] LustreError: 64734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786630378 with bad export cookie 9238187363270970407 [ 2296.537996] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2296.541425] LustreError: 64734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2296.844079] Lustre: server umount lustre-MDT0001 complete [ 2303.157591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2306.733767] Lustre: DEBUG MARKER: files compared the same [ 2319.166819] Lustre: server umount lustre-OST0000 complete [ 2337.298813] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 2340.488263] Key type lgssc unregistered [ 2340.753725] LNet: 74477:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2340.768718] LNetError: 74477:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2340.803334] LNet: Removed LNI 192.168.202.132@tcp [ 2341.723885] Key type .llcrypt unregistered [ 2341.727780] Key type ._llcrypt unregistered [ 2365.357958] Key type ._llcrypt registered [ 2365.359985] Key type .llcrypt registered [ 2365.484881] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 2366.474662] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2366.609965] alg: No test for adler32 (adler32-zlib) [ 2367.634312] Lustre: Lustre: Build Version: 2.17.57_2_g7a6d6aa [ 2367.885640] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 2369.552143] Key type lgssc registered [ 2370.558513] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2377.281081] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2377.290702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2378.656064] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2383.164231] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug +malloc all [ 2388.960854] LustreError: 75896:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2388.968515] LustreError: 75896:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2392.480679] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2392.745429] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2397.799884] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug +malloc all [ 2405.895980] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2411.667787] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2417.611834] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2417.890904] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2423.018358] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2423.953706] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug +malloc all [ 2430.940812] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2443.746478] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2443.760719] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2443.770744] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2447.841198] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2447.843272] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2447.849201] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2449.592777] Lustre: server umount lustre-OST0000 complete [ 2452.959750] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2452.963553] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2452.967487] Lustre: Skipped 1 previous similar message [ 2452.976667] Lustre: Skipped 2 previous similar messages [ 2458.079898] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2458.086940] Lustre: Skipped 1 previous similar message [ 2463.207450] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2463.218574] Lustre: Skipped 1 previous similar message [ 2467.295215] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2467.601771] Lustre: server umount lustre-MDT0000 complete [ 2471.329487] LustreError: 75877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786630553 with bad export cookie 767712939515687951 [ 2471.331142] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2471.346897] LustreError: 75877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2471.608319] Lustre: server umount lustre-MDT0001 complete [ 2482.067695] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 2484.998851] Key type lgssc unregistered [ 2485.314876] LNet: 79451:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2485.323739] LNetError: 79451:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2485.339369] LNet: Removed LNI 192.168.202.132@tcp [ 2486.244689] Key type .llcrypt unregistered [ 2486.246549] Key type ._llcrypt unregistered [ 2509.635899] Key type ._llcrypt registered [ 2509.639249] Key type .llcrypt registered [ 2509.783085] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 2511.047905] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2511.187812] alg: No test for adler32 (adler32-zlib) [ 2512.237200] Lustre: Lustre: Build Version: 2.17.57_2_g7a6d6aa [ 2512.478870] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 2514.127214] Key type lgssc registered [ 2515.524335] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2523.503455] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2523.516879] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2539.039866] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2544.159302] LustreError: 80841:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.132@tcp: failed processing log, type 4: rc = -110 [ 2570.847285] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2577.759948] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2585.601903] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2593.134459] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2593.248953] Lustre: *** cfs_fail_loc=706, val=0*** [ 2593.820866] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2593.917431] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2598.887328] LustreError: 80867:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2598.926491] LustreError: 80867:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2599.234416] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2604.008973] LustreError: 80866:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2605.029020] LustreError: 81926:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2605.044366] LustreError: 81926:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2608.456379] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2608.870091] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2614.552700] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2622.771555] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2630.230942] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2634.723828] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2634.728717] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2634.761433] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2638.749865] Lustre: server umount lustre-OST0000 complete [ 2644.962095] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2644.967167] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2644.973591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2644.973598] Lustre: Skipped 1 previous similar message [ 2644.996050] Lustre: Skipped 2 previous similar messages [ 2649.631545] Lustre: server umount lustre-MDT0000 complete [ 2650.080439] LustreError: 81930:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2650.098656] LustreError: 81930:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2653.787617] LustreError: 80849:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786630735 with bad export cookie 15429308685858901180 [ 2653.809280] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2654.395745] Lustre: server umount lustre-MDT0001 complete [ 2666.241821] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 2669.324502] Key type lgssc unregistered [ 2669.705692] LNet: 84458:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2669.731154] LNetError: 84458:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2669.747085] LNet: Removed LNI 192.168.202.132@tcp [ 2670.707765] Key type .llcrypt unregistered [ 2670.712075] Key type ._llcrypt unregistered [ 2697.846952] Key type ._llcrypt registered [ 2697.851152] Key type .llcrypt registered [ 2697.955621] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 2699.237489] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2699.337493] alg: No test for adler32 (adler32-zlib) [ 2700.422968] Lustre: Lustre: Build Version: 2.17.57_2_g7a6d6aa [ 2700.627372] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 2702.367881] Key type lgssc registered [ 2703.776778] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2712.996389] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2713.010890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2719.885440] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2729.298672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2730.248962] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2735.604315] LustreError: 86577:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2735.628108] LustreError: 86577:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2737.579319] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2740.714722] LustreError: 86580:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2740.771123] LustreError: 86580:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2745.826731] LustreError: 86581:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2745.858834] LustreError: 86581:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2750.842703] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2750.944460] LustreError: 86580:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2750.976072] LustreError: 86580:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2751.828957] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2755.953149] LustreError: 86581:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2755.987588] LustreError: 86581:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 2761.412508] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2766.306500] LustreError: 86575:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2766.338553] LustreError: 86575:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2771.557671] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2771.573129] Lustre: MGS: Client b83d5543-09f8-4a47-b920-1d38e3592bab (at 0@lo) reconnecting [ 2771.591578] LustreError: 86591:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2771.765433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2771.870459] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2777.047653] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2793.957130] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2793.963321] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2793.992202] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2797.536938] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2797.539077] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2797.555129] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2799.016599] Lustre: server umount lustre-OST0000 complete [ 2807.777218] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2807.781485] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2807.799119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2807.799126] Lustre: Skipped 1 previous similar message [ 2807.837416] Lustre: Skipped 1 previous similar message [ 2809.672035] Lustre: server umount lustre-MDT0000 complete [ 2812.897155] LustreError: 86581:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2812.922655] LustreError: 86581:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 2814.236541] Lustre: server umount lustre-MDT0001 complete [ 2818.446818] Lustre: server umount MGS complete [ 2830.908368] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 10:21:50 (1786630910) [ 2862.375094] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [ 2871.994919] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 2911.456858] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2917.055289] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2927.291543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2927.493637] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2927.528122] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2927.663475] Lustre: lustre-MDT0001: new disk, initializing [ 2927.724611] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2927.736298] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2927.739511] Lustre: 93775:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2927.744228] LustreError: 93775:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2932.169363] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2945.485282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2945.801053] Lustre: lustre-OST0000: new disk, initializing [ 2945.809093] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2945.815646] Lustre: 94541:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2945.912304] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2953.733821] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2954.288453] Lustre: 94855:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@0000000008b2c2a0 x1873418158910848/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1786631052 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2965.341710] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2972.741240] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2972.759376] Lustre: MGS: Client 133c8fc4-125c-40e4-b82a-28838a8aeb08 (at 0@lo) reconnecting [ 2972.830926] Lustre: 95587:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2972.857509] LustreError: 93761:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2972.876729] LustreError: 93761:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2972.902710] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2973.042115] Lustre: lustre-MDT0000: new disk, initializing [ 2973.219658] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2973.247503] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2973.292790] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2974.370609] Lustre: 95611:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@000000002d351781 x1873418158917888/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1786631072 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2978.526914] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2979.491255] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2981.871837] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2981.883746] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2983.542583] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 2983.544844] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 2983.548791] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 3009.505338] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3009.521655] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3009.539700] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3009.542279] Lustre: Skipped 1 previous similar message [ 3017.704376] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3017.708939] Lustre: Skipped 3 previous similar messages [ 3023.327158] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3023.527042] Lustre: server umount lustre-OST0000 complete [ 3033.057560] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3033.066260] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3033.097199] Lustre: Skipped 2 previous similar messages [ 3033.123769] Lustre: Skipped 3 previous similar messages [ 3035.096837] Lustre: server umount lustre-MDT0000 complete [ 3038.184208] LustreError: 93747:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3038.213602] LustreError: 93747:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3046.410357] Lustre: server umount lustre-MDT0001 complete [ 3051.589366] Lustre: server umount MGS complete [ 3065.976578] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 10:25:45 (1786631145) [ 3084.870631] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 3088.635860] Key type lgssc unregistered [ 3089.016639] LNet: 98123:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3089.028447] LNetError: 98123:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3090.085056] LNet: Removed LNI 192.168.202.132@tcp [ 3091.339977] Key type .llcrypt unregistered [ 3091.343510] Key type ._llcrypt unregistered [ 3111.898400] Key type ._llcrypt registered [ 3111.899955] Key type .llcrypt registered [ 3112.051612] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 3114.390646] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3114.501266] alg: No test for adler32 (adler32-zlib) [ 3115.865608] Lustre: Lustre: Build Version: 2.17.57_2_g7a6d6aa [ 3116.453532] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 3118.199184] Key type lgssc registered [ 3120.326837] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3130.403288] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3130.422808] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3131.906682] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3137.607520] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3141.092470] LustreError: 99347:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3141.142362] LustreError: 99347:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3146.016475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3146.229541] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3151.521411] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3161.481990] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3169.275244] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3177.709282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3178.237753] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3186.509194] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3195.133745] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3201.839603] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 3217.379506] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3217.386384] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3217.388060] LustreError: Skipped 1 previous similar message [ 3217.389484] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3217.396267] Lustre: Skipped 1 previous similar message [ 3217.407169] Lustre: Skipped 1 previous similar message [ 3220.529420] Lustre: server umount lustre-OST0000 complete [ 3227.618683] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3227.637153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3232.735991] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3232.741270] Lustre: Skipped 2 previous similar messages [ 3237.858807] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3237.864913] Lustre: Skipped 1 previous similar message [ 3238.367120] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3238.521744] Lustre: server umount lustre-MDT0000 complete [ 3241.807581] LustreError: 99328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786631323 with bad export cookie 6200752778081055576 [ 3241.810244] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3241.820030] LustreError: 99328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3242.154171] Lustre: server umount lustre-MDT0001 complete [ 3253.823102] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 3256.530733] Key type lgssc unregistered [ 3256.909375] LNet: 102907:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3256.929685] LNetError: 102907:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3256.969192] LNet: Removed LNI 192.168.202.132@tcp [ 3257.719250] Key type .llcrypt unregistered [ 3257.720771] Key type ._llcrypt unregistered [ 3279.749707] Key type ._llcrypt registered [ 3279.753115] Key type .llcrypt registered [ 3279.973929] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 3281.528315] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3281.561782] alg: No test for adler32 (adler32-zlib) [ 3282.784984] Lustre: Lustre: Build Version: 2.17.57_2_g7a6d6aa [ 3283.034049] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 3284.871228] Key type lgssc registered [ 3286.007112] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3292.892650] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3292.893830] LustreError: 103992:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3292.907292] Lustre: Skipped 1 previous similar message [ 3292.907317] LustreError: 103993:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3292.941552] LustreError: 103992:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=5000 [ 3292.947590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3292.950562] LustreError: 103992:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3292.950589] LustreError: 103992:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3294.469051] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3302.675856] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3302.986078] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3307.745659] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3314.731782] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing lsmod [ 3318.794125] LustreError: 105197:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3318.884188] LustreError: 105201:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3318.896937] LustreError: 105197:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4918 [ 3318.908090] LustreError: 105197:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3318.916779] LustreError: 105197:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3318.937408] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3319.334064] LustreError: 105212:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3319.352505] LustreError: 105212:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3319.556131] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3324.898122] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3324.900458] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3324.917083] Lustre: Skipped 2 previous similar messages [ 3324.922571] Lustre: Skipped 2 previous similar messages [ 3327.975909] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3327.981796] Lustre: Skipped 2 previous similar messages [ 3330.494805] Lustre: server umount lustre-MDT0000 complete [ 3333.089922] LustreError: 104032:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3333.127258] LustreError: 104032:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3335.294188] LustreError: 104004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786631417 with bad export cookie 12673830697841363977 [ 3335.322778] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3336.223229] LustreError: 104461:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000007a93aa35 x1873418770208128/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3336.456675] Lustre: server umount lustre-MDT0001 complete [ 3351.528474] Lustre: server umount lustre-OST0000 complete [ 3361.742754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3362.220139] LustreError: 106266:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3362.302579] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3367.124723] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3367.396582] LustreError: 106267:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3372.521276] LustreError: 106266:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3377.537750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3377.640378] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3377.855371] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3382.775660] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3390.934280] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3397.217501] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3404.827758] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3405.275956] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3407.289311] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 3413.065144] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3422.598529] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3432.932364] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3432.960528] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3432.968041] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3436.634436] Lustre: server umount lustre-OST0000 complete [ 3443.171287] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3443.183831] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3443.201747] Lustre: Skipped 2 previous similar messages [ 3443.240082] Lustre: Skipped 2 previous similar messages [ 3448.296114] LustreError: 106263:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3448.321163] LustreError: 106263:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3448.502482] Lustre: server umount lustre-MDT0000 complete [ 3454.019983] LustreError: 106246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786631536 with bad export cookie 12673830697841364677 [ 3454.047625] LustreError: 106246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3454.082295] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3454.703424] Lustre: server umount lustre-MDT0001 complete [ 3469.847685] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 3473.697496] Key type lgssc unregistered [ 3474.077525] LNet: 109803:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3474.093724] LNetError: 109803:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3474.111979] LNet: Removed LNI 192.168.202.132@tcp [ 3475.150322] Key type .llcrypt unregistered [ 3475.152386] Key type ._llcrypt unregistered [ 3504.732455] Key type ._llcrypt registered [ 3504.740319] Key type .llcrypt registered [ 3504.858899] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 3506.657933] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3506.691127] alg: No test for adler32 (adler32-zlib) [ 3507.776357] Lustre: Lustre: Build Version: 2.17.57_2_g7a6d6aa [ 3508.136449] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 3509.864200] Key type lgssc registered [ 3511.377405] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3521.580171] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3521.622965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3523.139766] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3528.595930] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3533.291092] LustreError: 111221:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3533.308275] LustreError: 111221:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3538.409019] LustreError: 111222:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3538.566325] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3538.973152] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3544.201297] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3552.982306] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3560.723542] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3569.685604] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3570.140675] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3577.256995] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3586.838732] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3593.746763] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:67 to 0x240000401:97) [ 3602.736729] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3606.388887] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3614.177668] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3614.178701] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3614.192997] LustreError: Skipped 1 previous similar message [ 3614.211132] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3614.225389] Lustre: Skipped 1 previous similar message [ 3617.312254] Lustre: server umount lustre-OST0000 complete [ 3624.421113] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3624.431641] Lustre: Skipped 1 previous similar message [ 3624.435566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3624.447783] Lustre: Skipped 1 previous similar message [ 3627.876646] Lustre: server umount lustre-MDT0000 complete [ 3629.537295] LustreError: 112189:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3629.550356] LustreError: 112189:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3631.809198] LustreError: 111202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786631713 with bad export cookie 242432436377927964 [ 3631.821097] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3631.841495] LustreError: 111202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3632.504660] Lustre: server umount lustre-MDT0001 complete [ 3646.111102] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 3650.042854] Key type lgssc unregistered [ 3650.367610] LNet: 114885:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3650.378559] LNetError: 114885:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3650.402518] LNet: Removed LNI 192.168.202.132@tcp [ 3651.356513] Key type .llcrypt unregistered [ 3651.358951] Key type ._llcrypt unregistered [ 3673.217390] Key type ._llcrypt registered [ 3673.219213] Key type .llcrypt registered [ 3673.341961] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 3674.795054] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3675.092544] alg: No test for adler32 (adler32-zlib) [ 3676.156782] Lustre: Lustre: Build Version: 2.17.57_2_g7a6d6aa [ 3676.476815] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 3678.143150] Key type lgssc registered [ 3679.228747] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3687.100585] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3687.117963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3688.547350] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3692.379823] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3698.166285] LustreError: 116110:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3698.185545] LustreError: 116110:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3700.285753] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3700.587211] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3705.113865] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3712.712374] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3719.059618] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3729.364132] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3729.632628] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3733.764106] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 3737.749243] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3746.534638] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3755.490860] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3755.496645] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3755.507937] Lustre: Skipped 1 previous similar message [ 3755.509216] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3755.537692] Lustre: Skipped 1 previous similar message [ 3760.614643] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3760.628183] Lustre: Skipped 1 previous similar message [ 3761.546563] Lustre: server umount lustre-OST0000 complete [ 3767.267972] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3767.275895] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3767.307400] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3770.851733] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3770.855650] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3772.356584] Lustre: server umount lustre-MDT0000 complete [ 3775.968497] LustreError: 116111:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3775.986083] LustreError: 116111:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3776.441650] LustreError: 116091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786631858 with bad export cookie 4701318778895938605 [ 3776.462598] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3777.011306] Lustre: server umount lustre-MDT0001 complete [ 3788.786816] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 3792.641534] Key type lgssc unregistered [ 3793.039705] LNet: 119647:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3793.048577] LNetError: 119647:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3793.070890] LNet: Removed LNI 192.168.202.132@tcp [ 3794.062235] Key type .llcrypt unregistered [ 3794.064674] Key type ._llcrypt unregistered [ 3820.640499] Key type ._llcrypt registered [ 3820.643475] Key type .llcrypt registered [ 3820.799550] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 3822.201927] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3822.448717] alg: No test for adler32 (adler32-zlib) [ 3823.725632] Lustre: Lustre: Build Version: 2.17.57_2_g7a6d6aa [ 3824.167196] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 3825.928420] Key type lgssc registered [ 3827.725767] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3837.837437] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3837.860540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3839.448884] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3843.911219] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3849.708878] LustreError: 121066:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3849.732692] LustreError: 121066:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3854.510884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3854.906197] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3860.301790] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3869.471284] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3875.554473] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3883.277391] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3883.566150] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3884.636395] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 3889.644847] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3898.056488] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3907.997336] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3912.524486] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3912.529910] Lustre: Skipped 2 previous similar messages [ 3915.385766] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3915.419342] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3917.116224] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3917.119817] Lustre: Skipped 2 previous similar messages [ 3922.027876] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3922.038351] Lustre: Skipped 1 previous similar message [ 3928.641212] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.32@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3928.652801] Lustre: Skipped 2 previous similar messages [ 3931.083767] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.32@tcp 192.168.0.[2,10]@tcp [ 3940.970946] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.32/24@tcp in log lustre-MDT0000 [ 3940.978790] Lustre: Skipped 2 previous similar messages [ 3946.515217] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.32/24@tcp [ 3946.519440] Lustre: Skipped 1 previous similar message [ 3959.270179] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3959.275197] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3959.282883] Lustre: Skipped 1 previous similar message [ 3959.289339] Lustre: Skipped 1 previous similar message [ 3963.683633] Lustre: server umount lustre-OST0000 complete [ 3969.505897] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3969.508521] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3969.532499] Lustre: Skipped 1 previous similar message [ 3969.543803] Lustre: Skipped 1 previous similar message [ 3974.624337] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3974.631606] Lustre: Skipped 1 previous similar message [ 3979.748335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3979.754041] Lustre: Skipped 1 previous similar message [ 3982.303149] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3982.441095] Lustre: server umount lustre-MDT0000 complete [ 3984.864709] LustreError: 121065:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3984.882953] LustreError: 121065:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3986.372550] LustreError: 121046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786632068 with bad export cookie 212718634675034102 [ 3986.380413] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3986.388480] LustreError: 121046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3986.851538] Lustre: server umount lustre-MDT0001 complete [ 3999.667460] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 4002.073751] Key type lgssc unregistered [ 4002.334544] LNet: 125965:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4002.337650] LNetError: 125965:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4002.353696] LNet: Removed LNI 192.168.202.132@tcp [ 4003.300678] Key type .llcrypt unregistered [ 4003.302340] Key type ._llcrypt unregistered [ 4034.002482] Key type ._llcrypt registered [ 4034.004800] Key type .llcrypt registered [ 4034.097895] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 4035.310095] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4035.513984] alg: No test for adler32 (adler32-zlib) [ 4036.771233] Lustre: Lustre: Build Version: 2.17.57_2_g7a6d6aa [ 4037.240733] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 4039.139153] Key type lgssc registered [ 4040.430934] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4048.904459] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4048.916858] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4050.389126] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 4050.429303] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.32/24@tcp [ 4050.441769] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4055.578963] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 4060.652915] LustreError: 127535:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4060.700453] LustreError: 127535:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4065.769691] LustreError: 127534:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4065.903131] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4066.425306] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4066.473259] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.32/24@tcp [ 4066.563119] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4071.371822] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 4079.834334] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4087.191853] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4094.930781] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4095.316830] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4096.374936] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:169 to 0x240000401:193) [ 4102.181757] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 4111.327774] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4125.700651] Lustre: server umount lustre-OST0000 complete [ 4132.320605] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4132.345923] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4137.445544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4137.453333] Lustre: Skipped 2 previous similar messages [ 4140.601227] Lustre: server umount lustre-MDT0000 complete [ 4142.563387] LustreError: 127534:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4142.598333] LustreError: 127534:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4143.585925] Lustre: 126842:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786632209/real 1786632209] req@000000007ae32d75 x1873419560844032/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786632225 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4143.611121] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4143.631481] Lustre: Skipped 1 previous similar message [ 4145.584691] LustreError: 127516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786632227 with bad export cookie 10266294224313852983 [ 4145.605088] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4146.055828] Lustre: server umount lustre-MDT0001 complete [ 4162.029974] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 4165.161307] Key type lgssc unregistered [ 4165.577272] LNet: 131127:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4165.590555] LNetError: 131127:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4165.619653] LNet: Removed LNI 192.168.202.132@tcp [ 4166.746898] Key type .llcrypt unregistered [ 4166.748751] Key type ._llcrypt unregistered [ 4199.059534] Key type ._llcrypt registered [ 4199.063474] Key type .llcrypt registered [ 4199.275615] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [ 4216.856373] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 4217.958774] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4218.073169] alg: No test for adler32 (adler32-zlib) [ 4219.370253] Lustre: Lustre: Build Version: 2.17.57_2_g7a6d6aa [ 4219.677786] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 4221.447171] Key type lgssc registered [ 4222.785439] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4264.110072] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4264.175387] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4265.504380] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4265.528773] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4265.607469] Lustre: lustre-MDT0000: new disk, initializing [ 4265.706465] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4265.721264] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4271.854915] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 4283.684269] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4283.817808] Lustre: 134530:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4283.854679] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4283.858295] Lustre: Skipped 1 previous similar message [ 4283.963646] Lustre: lustre-MDT0001: new disk, initializing [ 4284.058075] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4284.089943] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4284.093825] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4288.946669] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 4300.555263] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4307.160484] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4314.492717] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4314.927482] Lustre: lustre-OST0000: new disk, initializing [ 4314.931448] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4314.954789] Lustre: 136500:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4315.062485] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4322.294286] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 4323.928635] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4323.941568] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4324.015884] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4333.006825] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4338.372413] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4338.614549] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4344.175293] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4344.422069] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4349.930215] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4349.950958] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4349.955178] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4349.959114] Lustre: Skipped 1 previous similar message [ 4349.972550] LustreError: Skipped 1 previous similar message [ 4353.068612] Lustre: server umount lustre-OST0000 complete [ 4360.169417] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4360.182823] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4360.191904] Lustre: Skipped 1 previous similar message [ 4360.207522] Lustre: Skipped 2 previous similar messages [ 4363.947680] Lustre: server umount lustre-MDT0000 complete [ 4368.244093] LustreError: 134522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786632450 with bad export cookie 16918270288224185535 [ 4368.253562] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4368.255634] LustreError: 134522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4368.570848] Lustre: server umount lustre-MDT0001 complete [ 4377.086173] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4377.625118] LustreError: 138717:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4377.639994] LustreError: 138717:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4377.711958] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4382.919202] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 4383.228740] LustreError: 138718:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4387.298771] LustreError: 138717:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4392.417748] LustreError: 138718:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4392.570598] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4393.126275] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4398.508312] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 4408.080666] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4414.640823] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4422.117432] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4422.466913] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4429.027685] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 4438.919493] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4445.073558] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4447.726506] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 4455.027630] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4455.263124] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4466.271415] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4466.687508] Lustre: lustre-OST0001: new disk, initializing [ 4466.694362] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4466.709281] Lustre: 142105:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4466.881275] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4474.055821] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4474.068853] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4474.191212] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4474.795400] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 4484.516120] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4484.822719] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4490.149649] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4490.491563] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4496.842080] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9bdd88231800.ost_server_uuid 50 [ 4498.429919] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9bdd88231800.ost_server_uuid in IDLE state after 0 sec [ 4509.929055] Lustre: server umount lustre-OST0001 complete [ 4510.213496] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4510.232797] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4510.236559] LustreError: 140541:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4510.283238] LustreError: 140541:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4515.300612] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4515.316652] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4515.320560] Lustre: Skipped 2 previous similar messages [ 4515.336506] Lustre: Skipped 1 previous similar message [ 4520.420861] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4520.430939] LustreError: 140539:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4520.436694] Lustre: Skipped 1 previous similar message [ 4520.462235] LustreError: 140539:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4525.545434] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4525.558507] Lustre: Skipped 1 previous similar message [ 4529.121261] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4529.343056] Lustre: server umount lustre-OST0000 complete [ 4535.777733] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4535.783788] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4535.791201] Lustre: Skipped 1 previous similar message [ 4535.801620] Lustre: Skipped 1 previous similar message [ 4539.993239] Lustre: server umount lustre-MDT0000 complete [ 4540.896863] LustreError: 138714:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4540.923282] LustreError: 138714:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4544.434362] LustreError: 138698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786632626 with bad export cookie 16918270288224186417 [ 4544.441655] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4544.446232] LustreError: 138698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4544.857983] Lustre: server umount lustre-MDT0001 complete [ 4567.986619] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 4571.234645] Key type lgssc unregistered [ 4571.678937] LNet: 145110:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4571.702677] LNetError: 145110:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4571.726218] LNet: Removed LNI 192.168.202.132@tcp [ 4573.052107] Key type .llcrypt unregistered [ 4573.055333] Key type ._llcrypt unregistered [ 4622.941466] Key type ._llcrypt registered [ 4622.952303] Key type .llcrypt registered [ 4623.104340] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [ 4644.597533] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 4646.096338] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4646.170298] alg: No test for adler32 (adler32-zlib) [ 4647.318504] Lustre: Lustre: Build Version: 2.17.57_2_g7a6d6aa [ 4647.572963] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 4649.303741] Key type lgssc registered [ 4650.593187] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4692.322637] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4692.363382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4693.644739] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4693.697381] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4693.787505] Lustre: lustre-MDT0000: new disk, initializing [ 4693.881297] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4693.900791] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4699.199524] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 4712.053706] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4712.184899] Lustre: 149274:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4712.276881] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4712.284429] Lustre: Skipped 1 previous similar message [ 4712.354938] Lustre: lustre-MDT0001: new disk, initializing [ 4712.450835] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4712.511788] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4712.516897] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4717.552336] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 4728.969829] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4736.333599] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4745.958697] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4746.301341] Lustre: lustre-OST0000: new disk, initializing [ 4746.307131] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4746.313282] Lustre: 151243:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4746.389965] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4754.328589] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 4756.151571] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4756.180586] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4756.403988] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4767.011590] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4771.701287] Lustre: Failing over lustre-OST0000 [ 4771.847736] Lustre: server umount lustre-OST0000 complete [ 4787.169729] Lustre: 147498:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786632853/real 1786632853] req@0000000019eec680 x1873420200478592/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786632869 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4787.176077] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4787.207509] Lustre: 147498:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4788.191403] Lustre: 147499:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786632854/real 1786632854] req@000000009feeb869 x1873420200479104/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1786632870 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4792.287334] Lustre: 147498:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786632858/real 1786632858] req@0000000005890615 x1873420200480000/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786632874 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4793.052450] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4793.401168] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4793.419669] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4797.479199] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4798.507652] Lustre: 147499:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786632864/real 1786632864] req@000000001a7ce32a x1873420200481152/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786632880 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4798.534939] Lustre: 147499:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4800.521557] Lustre: lustre-OST0000: Recovery over after 0:03, of 3 clients 3 recovered and 0 were evicted. [ 4800.527784] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4800.943426] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 4803.552133] Lustre: 147500:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786632869/real 1786632869] req@0000000023b42f5c x1873420200482304/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786632885 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4803.601606] Lustre: 147500:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4805.325292] Lustre: Failing over lustre-MDT0000 [ 4805.653081] Lustre: server umount lustre-MDT0000 complete [ 4807.650793] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4807.682616] Lustre: Skipped 3 previous similar messages [ 4810.724580] LustreError: 149283:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4810.748106] LustreError: 149283:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 4812.768870] LustreError: 149288:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4812.796044] LustreError: 149288:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4815.838972] LustreError: 149283:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4817.888708] LustreError: 149288:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4817.942431] LustreError: 149288:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4823.007822] Lustre: 147499:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786632889/real 1786632889] req@0000000023b42f5c x1873420200492416/t0(0) o400->MGC192.168.202.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786632905 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4823.012850] LustreError: 149287:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4823.053411] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4823.103864] LustreError: 149287:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 4825.333230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4831.204400] LustreError: 151934:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4831.220884] LustreError: 151934:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 4833.585147] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4833.638668] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4836.326817] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4838.705533] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 4838.898440] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4838.906347] Lustre: Skipped 1 previous similar message [ 4838.974387] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 4839.077775] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4849.126422] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4849.164175] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4854.246965] LustreError: 152481:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4854.256693] LustreError: 152481:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 4854.362656] Lustre: server umount lustre-OST0000 complete [ 4859.360850] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4859.374194] Lustre: Skipped 1 previous similar message [ 4859.383139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4859.397449] Lustre: Skipped 2 previous similar messages [ 4864.480611] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4864.490104] Lustre: Skipped 1 previous similar message [ 4865.665910] Lustre: server umount lustre-MDT0000 complete [ 4870.410819] LustreError: 149268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786632952 with bad export cookie 1010174277215347491 [ 4870.422064] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4870.427232] LustreError: 149268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4871.185528] Lustre: server umount lustre-MDT0001 complete [ 4884.711468] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 4888.362788] Key type lgssc unregistered [ 4888.841547] LNet: 154904:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4888.851939] LNetError: 154904:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4888.866373] LNet: Removed LNI 192.168.202.132@tcp [ 4889.682637] Key type .llcrypt unregistered [ 4889.685673] Key type ._llcrypt unregistered [ 4928.872837] Key type ._llcrypt registered [ 4928.876477] Key type .llcrypt registered [ 4928.989785] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 4930.907904] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4930.946195] alg: No test for adler32 (adler32-zlib) [ 4931.997440] Lustre: Lustre: Build Version: 2.17.57_2_g7a6d6aa [ 4932.288483] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 4933.991819] Key type lgssc registered [ 4935.708789] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4945.665673] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4945.682225] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4947.298419] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4952.477302] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 4957.686167] LustreError: 156430:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4957.709069] LustreError: 156430:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4962.791327] LustreError: 156429:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4963.936606] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4967.906164] LustreError: 156430:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4968.155189] LustreError: 156458:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4968.168234] LustreError: 156458:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4968.173473] LustreError: 156458:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 21, retries 0, failed: rc = -5 [ 4968.482583] Lustre: server umount lustre-MDT0000 complete [ 4985.733751] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [ 4997.842639] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 5039.754885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5040.157529] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5040.192899] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5040.318885] Lustre: lustre-MDT0000: new disk, initializing [ 5040.414692] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5040.428191] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5046.008886] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 5058.846913] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5059.056520] Lustre: 159763:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5059.113501] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5059.116614] Lustre: Skipped 1 previous similar message [ 5059.192455] Lustre: lustre-MDT0001: new disk, initializing [ 5059.371946] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5059.439685] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5059.455355] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5065.087898] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 5075.571255] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5081.624117] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5089.562201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5089.851328] Lustre: lustre-OST0000: new disk, initializing [ 5089.858699] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5089.866974] Lustre: 161730:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5089.967423] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5096.506754] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 5099.634862] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5099.662857] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5099.900086] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5108.044827] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5113.912294] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 5115.677075] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 10:59:55 (1786633195) [ 5381.606922] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5381.620197] Lustre: Skipped 2 previous similar messages [ 5381.623057] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5381.634243] Lustre: Skipped 2 previous similar messages [ 5386.059893] Lustre: server umount lustre-MDT0000 complete [ 5386.738086] LustreError: 159768:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5386.753406] LustreError: 159768:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5390.210821] LustreError: 159754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786633472 with bad export cookie 11630663872906962268 [ 5390.216344] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5390.219582] LustreError: 159754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5391.839791] LustreError: 159770:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5391.839842] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5391.880995] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5395.872289] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5396.592968] Lustre: server umount lustre-MDT0001 complete [ 5406.908272] Lustre: server umount lustre-OST0000 complete [ 5416.888604] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [ 5427.070916] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 5464.143159] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5464.494957] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5464.546836] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5464.695808] Lustre: lustre-MDT0000: new disk, initializing [ 5464.816113] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5464.852586] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5469.779952] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 5481.083700] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5481.159120] Lustre: 166427:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5481.212635] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5481.215385] Lustre: Skipped 1 previous similar message [ 5481.358748] Lustre: lustre-MDT0001: new disk, initializing [ 5481.523533] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5481.558502] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5481.572220] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5486.494427] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 5496.786187] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5503.279046] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5511.093145] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5511.548837] Lustre: lustre-OST0000: new disk, initializing [ 5511.551130] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5511.555193] Lustre: 168395:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5511.640638] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5519.461227] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5519.472535] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5519.663393] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5519.753373] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 5531.991609] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5538.889456] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5539.183866] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5544.792664] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5545.021444] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5550.563650] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5550.566970] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5550.575722] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5550.587182] Lustre: Skipped 1 previous similar message [ 5553.791961] Lustre: server umount lustre-OST0000 complete [ 5558.248851] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5558.263653] LustreError: Skipped 1 previous similar message [ 5558.270750] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5558.292870] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5558.295466] Lustre: Skipped 1 previous similar message [ 5564.660400] Lustre: server umount lustre-MDT0000 complete [ 5565.924238] LustreError: 166436:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5565.945788] LustreError: 166436:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5568.562345] LustreError: 167220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786633650 with bad export cookie 11630663872907121098 [ 5568.564269] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5568.576976] LustreError: 167220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5568.871429] Lustre: server umount lustre-MDT0001 complete [ 5577.784801] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 11:07:37 (1786633657) [ 5594.466475] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [ 5610.139408] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 5652.048935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5652.308379] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5652.344266] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5652.429383] Lustre: lustre-MDT0000: new disk, initializing [ 5652.565819] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5652.617733] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5658.012179] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 5669.261273] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5669.401823] Lustre: 173178:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5669.787316] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5675.257481] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 5685.546077] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5692.153478] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5699.811402] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5700.100145] Lustre: lustre-OST0000: new disk, initializing [ 5700.102965] Lustre: Skipped 1 previous similar message [ 5700.107401] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5700.111875] Lustre: Skipped 2 previous similar messages [ 5700.119364] Lustre: 175156:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5707.404988] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 5709.908146] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5709.918921] Lustre: Skipped 1 previous similar message [ 5709.930099] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5710.023502] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5718.557949] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5730.785864] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5730.794375] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5730.806855] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5730.806863] Lustre: Skipped 2 previous similar messages [ 5730.839725] Lustre: Skipped 2 previous similar messages [ 5731.994424] Lustre: server umount lustre-OST0000 complete [ 5741.035089] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5743.041429] Lustre: server umount lustre-MDT0000 complete [ 5746.157882] LustreError: 173184:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5746.179515] LustreError: 173184:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5747.258530] LustreError: 173168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786633829 with bad export cookie 11630663872907121980 [ 5747.263660] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5747.281430] LustreError: 173168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5747.864758] Lustre: server umount lustre-MDT0001 complete [ 5757.672474] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 11:10:37 (1786633837) [ 5776.236749] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [ 5788.127355] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 5829.497766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5829.928328] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5829.986160] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5830.118868] Lustre: lustre-MDT0000: new disk, initializing [ 5830.284099] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5830.291778] Lustre: Skipped 2 previous similar messages [ 5830.326186] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5835.512897] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 5846.405594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5846.526457] Lustre: 179478:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5846.797657] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5852.091163] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 5862.650820] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5868.427665] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5878.751057] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5879.333419] Lustre: 181453:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5886.157619] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5886.574107] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5890.048046] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 5902.559308] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5912.032271] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5912.036572] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5912.052605] LustreError: Skipped 1 previous similar message [ 5912.055755] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5912.055767] Lustre: Skipped 3 previous similar messages [ 5912.079027] Lustre: Skipped 2 previous similar messages [ 5917.811500] Lustre: server umount lustre-OST0000 complete [ 5923.809096] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5928.687873] Lustre: server umount lustre-MDT0000 complete [ 5932.520596] LustreError: 181356:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5932.543870] LustreError: 181356:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5933.698190] LustreError: 179469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786634015 with bad export cookie 11630663872907123065 [ 5933.707206] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5933.716765] LustreError: 179469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5934.236884] Lustre: server umount lustre-MDT0001 complete [ 5947.512933] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 5950.940961] Key type lgssc unregistered [ 5951.344786] LNet: 183317:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5951.353858] LNetError: 183317:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5951.365441] LNet: Removed LNI 192.168.202.132@tcp [ 5952.346349] Key type .llcrypt unregistered [ 5952.348498] Key type ._llcrypt unregistered [ 5981.610734] Key type ._llcrypt registered [ 5981.614125] Key type .llcrypt registered [ 5981.785884] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 5983.366185] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5983.437476] alg: No test for adler32 (adler32-zlib) [ 5984.513751] Lustre: Lustre: Build Version: 2.17.57_2_g7a6d6aa [ 5984.734414] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 5986.415180] Key type lgssc registered [ 5987.675659] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5996.120650] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5996.147165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5997.647967] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6002.864922] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6007.781622] LustreError: 184740:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6007.825655] LustreError: 184740:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6012.904867] LustreError: 184739:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6013.027134] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6013.535462] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6018.844519] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6027.727308] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6034.265706] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6043.146438] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6043.680789] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6050.144573] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6058.139715] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6067.254215] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 6087.648723] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6087.661524] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6087.671758] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6087.679026] Lustre: Skipped 1 previous similar message [ 6087.700085] Lustre: Skipped 1 previous similar message [ 6092.779047] LustreError: 186562:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6092.804873] LustreError: 186562:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6093.078768] Lustre: server umount lustre-OST0000 complete [ 6097.896942] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6097.917699] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6097.929869] Lustre: Skipped 1 previous similar message [ 6103.009690] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6103.016106] Lustre: Skipped 1 previous similar message [ 6104.268941] Lustre: server umount lustre-MDT0000 complete [ 6108.129355] LustreError: 184739:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6108.150801] LustreError: 184739:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6108.171447] LustreError: 184720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786634190 with bad export cookie 5596639553851728233 [ 6108.182816] LustreError: 184720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6108.187538] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6108.928359] Lustre: server umount lustre-MDT0001 complete [ 6120.305033] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 6123.945970] Key type lgssc unregistered [ 6124.315118] LNet: 188300:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6124.324552] LNetError: 188300:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6125.349477] LNet: Removed LNI 192.168.202.132@tcp [ 6126.607149] Key type .llcrypt unregistered [ 6126.610926] Key type ._llcrypt unregistered [ 6152.880964] Key type ._llcrypt registered [ 6152.884118] Key type .llcrypt registered [ 6152.984822] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 6154.048760] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6154.197824] alg: No test for adler32 (adler32-zlib) [ 6155.254351] Lustre: Lustre: Build Version: 2.17.57_2_g7a6d6aa [ 6155.522303] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 6157.154316] Key type lgssc registered [ 6158.384857] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6166.052640] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6166.087631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6167.637609] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6172.730296] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6177.761910] LustreError: 189721:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6177.784579] LustreError: 189721:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6182.690392] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6182.885602] LustreError: 189722:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6183.166329] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6188.403703] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6198.046760] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6204.828201] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6213.232126] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6213.648415] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6220.893207] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6230.588632] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6237.244728] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6242.278189] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6242.283159] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6242.298430] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6242.329972] Lustre: Skipped 1 previous similar message [ 6242.829498] Lustre: lustre-OST0000: Not available for connect from 192.168.202.32@tcp (stopping) [ 6242.836961] Lustre: Skipped 1 previous similar message [ 6247.395418] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6247.405110] Lustre: Skipped 1 previous similar message [ 6252.516411] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6252.528236] Lustre: Skipped 2 previous similar messages [ 6256.095170] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6256.248941] Lustre: server umount lustre-OST0000 complete [ 6265.412944] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9bdd85176800.ost_server_uuid 50 [ 6267.373578] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6285.797726] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6285.811697] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6285.834643] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6285.850324] Lustre: Skipped 1 previous similar message [ 6287.329285] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6289.208362] Lustre: server umount lustre-MDT0000 complete [ 6292.461049] LustreError: 189718:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6292.486099] LustreError: 189718:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6294.102958] LustreError: 189703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786634376 with bad export cookie 4860296010597712724 [ 6294.112912] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6294.119655] LustreError: 189703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6294.476393] Lustre: server umount lustre-MDT0001 complete [ 6304.318658] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 11:19:43 (1786634383) [ 6311.875296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6312.415768] LustreError: 193644:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6312.522251] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6316.852473] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6317.546398] LustreError: 193645:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6327.175106] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6327.757411] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6333.394535] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6341.770769] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6348.677964] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6356.858939] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6357.330979] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6358.386712] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6365.584047] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6375.213448] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6382.693543] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6382.918481] Lustre: lustre-OST0001: new disk, initializing [ 6382.921715] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6382.926706] Lustre: 196484:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6383.034302] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6389.345993] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6389.358921] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6389.502140] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6389.986243] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6401.231638] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6404.580044] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6404.598611] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6404.616931] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6404.619601] Lustre: Skipped 2 previous similar messages [ 6410.431218] Lustre: server umount lustre-OST0000 complete [ 6412.266975] LustreError: 195466:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6412.278972] LustreError: 195466:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6416.894284] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6417.212595] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6423.062679] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6423.411941] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6430.719916] LustreError: 195464:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6430.740076] LustreError: 195464:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 6440.930771] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6440.930896] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6440.934409] LustreError: Skipped 1 previous similar message [ 6440.936058] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6440.936064] Lustre: Skipped 5 previous similar messages [ 6440.979714] Lustre: Skipped 2 previous similar messages [ 6443.957839] Lustre: server umount lustre-OST0001 complete [ 6450.662454] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6450.668112] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6454.794129] Lustre: server umount lustre-MDT0000 complete [ 6458.898365] LustreError: 193624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786634540 with bad export cookie 4860296010597713795 [ 6458.899861] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6458.907863] LustreError: 193624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6459.274371] Lustre: server umount lustre-MDT0001 complete [ 6486.642317] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 11:22:46 (1786634566) [ 6495.227239] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6495.795928] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6495.890941] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6496.354121] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6503.700694] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6514.564543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6514.655651] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6514.705411] Lustre: 199925:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6520.371483] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6530.016532] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6537.419480] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6544.095443] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6544.289648] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6551.235061] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 6551.972197] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6563.245566] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6570.895179] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6571.047446] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6571.289659] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6571.309823] Lustre: Skipped 2 previous similar messages [ 6579.096347] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6579.466630] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6587.597834] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6592.483131] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6592.485669] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6592.492919] LustreError: Skipped 1 previous similar message [ 6592.498211] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6592.498219] Lustre: Skipped 4 previous similar messages [ 6592.538954] Lustre: Skipped 2 previous similar messages [ 6597.006530] Lustre: server umount lustre-OST0000 complete [ 6598.637890] LustreError: 201774:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6598.660736] LustreError: 201774:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 6615.529926] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6619.495499] Lustre: server umount lustre-OST0001 complete [ 6625.760101] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6625.768218] Lustre: Skipped 3 previous similar messages [ 6631.864563] Lustre: server umount lustre-MDT0000 complete [ 6636.057716] LustreError: 199916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786634718 with bad export cookie 4860296010597715202 [ 6636.058884] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6636.066224] LustreError: 199916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6636.555836] Lustre: server umount lustre-MDT0001 complete [ 6663.702656] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 11:25:43 (1786634743) [ 6683.358311] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [ 6696.029369] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 6737.575438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6737.962813] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6738.035398] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6738.110965] Lustre: lustre-MDT0000: new disk, initializing [ 6738.231940] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6738.258339] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6743.489349] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6756.882432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6757.105445] Lustre: 208028:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6757.237191] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6757.244956] Lustre: Skipped 1 previous similar message [ 6757.377753] Lustre: lustre-MDT0001: new disk, initializing [ 6757.585124] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6757.620245] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6763.588395] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6774.098477] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6781.156676] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6789.639091] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6789.922916] Lustre: lustre-OST0000: new disk, initializing [ 6789.928877] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6789.934636] Lustre: 209998:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6797.872187] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6800.046286] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6800.057287] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6800.158696] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6808.651741] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6814.645773] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6814.976535] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6821.013399] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6821.355534] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6825.952912] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6825.954978] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6825.960572] LustreError: Skipped 1 previous similar message [ 6825.964923] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6825.983566] Lustre: Skipped 1 previous similar message [ 6826.009585] Lustre: Skipped 9 previous similar messages [ 6838.239117] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6838.459810] Lustre: server umount lustre-OST0000 complete [ 6848.798613] Lustre: server umount lustre-MDT0000 complete [ 6851.564791] LustreError: 208034:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6851.602213] LustreError: 208034:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 15 previous similar messages [ 6852.747627] LustreError: 208019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786634934 with bad export cookie 4860296010597716987 [ 6852.754615] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6852.762606] LustreError: 208019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6853.221972] Lustre: server umount lustre-MDT0001 complete [ 6863.085779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6868.996922] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6878.353445] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6884.375600] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6893.433820] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6900.433823] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6907.787555] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6916.488712] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6926.109945] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6931.807686] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6932.127499] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6938.899397] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6939.223139] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6944.226882] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6944.242534] LustreError: Skipped 1 previous similar message [ 6947.964344] Lustre: server umount lustre-OST0000 complete [ 6956.712372] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6963.917777] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6996.071022] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6996.568573] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6996.584428] Lustre: Skipped 5 previous similar messages [ 7005.423850] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7009.914097] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7009.932359] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7009.932740] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7009.962203] Lustre: Skipped 1 previous similar message [ 7013.957947] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7015.696657] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7025.643382] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7025.667541] Lustre: Skipped 5 previous similar messages [ 7025.671215] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7025.698558] Lustre: Skipped 10 previous similar messages [ 7027.453335] Lustre: server umount lustre-OST0000 complete [ 7038.472254] Lustre: server umount lustre-MDT0000 complete [ 7041.002931] LustreError: 212219:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7041.045606] LustreError: 212219:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 7043.416405] LustreError: 212199:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786635125 with bad export cookie 4860296010597717869 [ 7043.426128] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7043.429219] LustreError: 212199:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7043.848138] Lustre: server umount lustre-MDT0001 complete [ 7056.715525] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 11:32:15 (1786635135) [ 7065.544869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7071.588065] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7081.895506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7088.520746] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7097.115441] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7104.223470] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7111.707237] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7119.614794] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7128.641593] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7134.236693] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7136.782225] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 7142.571144] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7142.875974] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7150.675492] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7150.896640] Lustre: lustre-OST0001: new disk, initializing [ 7150.901785] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7150.908751] Lustre: 221580:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7158.283184] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7160.407416] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7160.418989] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7160.544476] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7170.302718] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7177.033415] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7177.414118] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7183.569826] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7183.997076] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7186.400208] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7186.408105] LustreError: Skipped 3 previous similar messages [ 7192.327643] Lustre: server umount lustre-OST0001 complete [ 7200.742513] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7201.003707] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7206.239247] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7206.501766] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7237.365264] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7242.773445] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7242.792874] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7242.794169] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7242.820544] Lustre: Skipped 1 previous similar message [ 7245.517570] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7253.903476] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7256.033613] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7264.545815] Lustre: server umount lustre-OST0001 complete [ 7283.690974] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7283.707782] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7283.716494] Lustre: Skipped 7 previous similar messages [ 7283.730391] Lustre: Skipped 10 previous similar messages [ 7288.663929] Lustre: server umount lustre-OST0000 complete [ 7299.504060] Lustre: server umount lustre-MDT0000 complete [ 7303.148266] LustreError: 218191:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7303.176330] LustreError: 218191:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 36 previous similar messages [ 7303.423698] LustreError: 218171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786635385 with bad export cookie 4860296010597718835 [ 7303.424038] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7303.439637] LustreError: 218171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7310.050832] Lustre: server umount lustre-MDT0001 complete [ 7338.042599] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 11:36:57 (1786635417) [ 7346.493106] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7346.941450] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7346.967328] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7352.093901] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7361.293788] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7361.364953] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7361.397074] Lustre: 226824:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7366.367330] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7374.520548] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7380.773991] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7387.794141] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7387.993981] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7394.597930] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7402.486506] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7411.931635] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7412.072408] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7419.134140] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7427.456922] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7429.236801] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7434.511463] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7434.867258] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7440.626597] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7440.910720] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7447.024137] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9bdd88232000.ost_server_uuid 50 [ 7448.864933] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9bdd88232000.ost_server_uuid in IDLE state after 0 sec [ 7449.956823] Lustre: Permanently deactivating lustre-OST0001 [ 7449.976571] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7454.040593] Lustre: Permanently reactivating lustre-OST0001 [ 7454.068063] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7454.081316] Lustre: Skipped 2 previous similar messages [ 7459.305748] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7459.307898] LustreError: Skipped 3 previous similar messages [ 7464.552793] Lustre: server umount lustre-OST0001 complete [ 7474.685313] Lustre: server umount lustre-OST0000 complete [ 7484.465350] Lustre: server umount lustre-MDT0000 complete [ 7488.463893] LustreError: 226819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786635570 with bad export cookie 4860296010597720123 [ 7488.481688] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7488.508839] LustreError: 226819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7489.002950] Lustre: server umount lustre-MDT0001 complete [ 7513.125976] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 11:39:52 (1786635592) [ 7521.721321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7522.186800] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7522.278537] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7522.304924] Lustre: Skipped 2 previous similar messages [ 7522.636108] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7522.645140] Lustre: Skipped 9 previous similar messages [ 7529.411240] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7539.113513] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7539.256821] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7539.303746] Lustre: 233521:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7545.186756] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7553.389526] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7559.490221] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7567.039646] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7567.293363] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7567.362480] Lustre: Permanently deactivating lustre-OST0000 [ 7567.383886] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7574.809626] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7576.618601] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7576.764697] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7583.350655] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7590.412939] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7590.468035] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7596.709911] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7597.403807] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7604.843511] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7606.938585] Lustre: Permanently reactivating lustre-OST0000 [ 7606.968261] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7606.974802] Lustre: Skipped 2 previous similar messages [ 7617.154590] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7617.161437] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7617.180732] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7617.224233] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7622.503574] Lustre: server umount lustre-OST0001 complete [ 7628.964970] Lustre: server umount lustre-OST0000 complete [ 7639.491760] Lustre: server umount lustre-MDT0000 complete [ 7643.677170] LustreError: 233513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786635725 with bad export cookie 4860296010597721789 [ 7643.709050] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7644.244482] Lustre: server umount lustre-MDT0001 complete [ 7657.602865] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 7660.725570] Key type lgssc unregistered [ 7661.116218] LNet: 238406:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7661.123062] LNetError: 238406:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7661.149827] LNet: Removed LNI 192.168.202.132@tcp [ 7661.946659] Key type .llcrypt unregistered [ 7661.948366] Key type ._llcrypt unregistered [ 7683.032790] Key type ._llcrypt registered [ 7683.036173] Key type .llcrypt registered [ 7683.158174] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 7684.442697] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7684.510923] alg: No test for adler32 (adler32-zlib) [ 7685.658876] Lustre: Lustre: Build Version: 2.17.57_2_g7a6d6aa [ 7685.995707] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 7687.696170] Key type lgssc registered [ 7688.732496] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7698.486700] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7698.512324] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7700.054533] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7705.791419] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7709.177858] LustreError: 239859:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7709.207374] LustreError: 239859:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7710.245707] LustreError: 239858:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7715.304297] LustreError: 239859:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7715.544792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7715.684630] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7715.721647] Lustre: 239847:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7715.729683] Lustre: Permanently deactivating lustre-MDT0001 [ 7715.740897] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7716.111818] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7720.679551] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7723.053329] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7728.602986] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7735.100561] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7742.705329] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7743.161311] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7744.199873] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7745.267362] LustreError: 241701:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7749.763910] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7750.636451] LustreError: 241701:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7759.818651] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7760.869033] LustreError: 241701:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7760.883491] LustreError: 241701:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7770.072404] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7770.512747] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7775.775935] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7775.812223] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7778.517684] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7788.355529] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7791.836170] Lustre: Permanently reactivating lustre-MDT0001 [ 7791.854218] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7791.871859] Lustre: Skipped 2 previous similar messages [ 7794.194736] Lustre: lustre-MDT0001: Client 350949b8-690f-4dd2-bc67-b4e102a4acf8 (at 192.168.202.32@tcp) reconnecting [ 7800.499415] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7800.512191] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7800.524703] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7800.554077] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7801.148658] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7803.136601] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7808.957969] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7809.200243] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7814.756789] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7814.980595] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7817.726329] Lustre: Permanently deactivating lustre-MDT0001 [ 7817.754795] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7817.757778] Lustre: Skipped 2 previous similar messages [ 7826.604959] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7837.164298] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7837.173695] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7837.176561] Lustre: Skipped 2 previous similar messages [ 7837.202640] Lustre: Skipped 3 previous similar messages [ 7841.253623] Lustre: server umount lustre-MDT0000 complete [ 7842.283932] LustreError: 242112:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7842.321962] LustreError: 242112:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 7845.292261] LustreError: 239839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786635927 with bad export cookie 10548903765761609029 [ 7845.294037] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7845.320753] LustreError: 239839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7845.649215] Lustre: server umount lustre-MDT0001 complete [ 7852.130738] Lustre: server umount lustre-OST0000 complete [ 7868.222256] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 11:45:47 (1786635947) [ 7881.575818] Lustre: server umount lustre-OST0001 complete [ 7888.370104] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [ 7899.482980] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 7937.848779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7938.071596] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7938.081829] Lustre: Skipped 2 previous similar messages [ 7938.107576] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7938.186565] Lustre: lustre-MDT0000: new disk, initializing [ 7938.252459] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7938.267267] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7943.455538] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7955.066808] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7955.235716] Lustre: 248173:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7955.286900] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7955.291100] Lustre: Skipped 1 previous similar message [ 7955.367307] Lustre: lustre-MDT0001: new disk, initializing [ 7955.521599] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7955.590772] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7955.614345] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7960.931493] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7971.980665] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7978.317827] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7985.707072] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7986.048920] Lustre: lustre-OST0000: new disk, initializing [ 7986.055466] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7986.060505] Lustre: 250143:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7986.164692] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7994.188644] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7995.580449] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7995.626593] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7995.885504] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8009.549477] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8012.595119] LustreError: 248179:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8022.615128] LustreError: 248179:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8022.668765] LustreError: 248178:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8023.062260] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8023.323717] Lustre: lustre-OST0001: new disk, initializing [ 8023.338701] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8023.348662] Lustre: 251258:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8023.453073] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8030.357459] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8030.382351] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8030.556514] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8030.656485] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8032.700620] LustreError: 248178:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8041.056741] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8042.777863] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8046.061276] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8046.067337] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8046.093145] Lustre: Skipped 1 previous similar message [ 8046.098541] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8046.104783] Lustre: Skipped 1 previous similar message [ 8051.181348] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8051.198380] Lustre: Skipped 1 previous similar message [ 8056.310627] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8056.320362] Lustre: Skipped 1 previous similar message [ 8059.359130] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8059.594398] Lustre: server umount lustre-OST0001 complete [ 8061.438552] LustreError: 250127:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8061.464901] LustreError: 250127:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8077.805092] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8077.806420] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8077.812171] LustreError: Skipped 2 previous similar messages [ 8077.813374] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8077.837177] Lustre: Skipped 2 previous similar messages [ 8083.439946] Lustre: server umount lustre-OST0000 complete [ 8088.544471] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8088.549975] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8088.567020] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8088.573169] Lustre: Skipped 3 previous similar messages [ 8093.896944] Lustre: server umount lustre-MDT0000 complete [ 8098.072828] LustreError: 248164:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786636180 with bad export cookie 10548903765761611626 [ 8098.074573] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8098.094442] LustreError: 248164:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8104.452039] Lustre: server umount lustre-MDT0001 complete [ 8117.451049] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 8121.637793] Key type lgssc unregistered [ 8122.048693] LNet: 253295:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8122.052729] LNetError: 253295:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8122.068832] LNet: Removed LNI 192.168.202.132@tcp [ 8123.068650] Key type .llcrypt unregistered [ 8123.077268] Key type ._llcrypt unregistered [ 8166.192954] Key type ._llcrypt registered [ 8166.195104] Key type .llcrypt registered [ 8166.308942] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 8167.662780] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8167.679583] alg: No test for adler32 (adler32-zlib) [ 8168.709737] Lustre: Lustre: Build Version: 2.17.57_2_g7a6d6aa [ 8169.066934] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 8170.839236] Key type lgssc registered [ 8172.493875] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8181.161721] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8181.197311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8182.507748] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8182.558096] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8182.738590] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8188.259661] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8197.915245] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8197.998220] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8198.034803] Lustre: 255503:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8198.295964] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8203.067727] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8211.292427] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8218.354770] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8226.334132] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8226.599716] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8226.815583] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8234.723327] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8236.220478] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8244.388864] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8300.642506] Lustre: server umount lustre-OST0000 complete [ 8306.337461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8318.368287] Lustre: 254828:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786636384/real 1786636384] req@000000009a9eb4d5 x1873423893574912/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786636400 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8318.404923] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8319.457482] Lustre: 254826:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786636385/real 1786636385] req@00000000bb2babb0 x1873423893575808/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786636401 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8319.465693] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8319.487728] Lustre: 254826:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 8323.169088] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8323.529638] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8324.066377] Lustre: 254825:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786636390/real 1786636390] req@000000007812c872 x1873423893576832/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786636406 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8328.688402] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8328.705363] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8329.705046] Lustre: 254827:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786636395/real 1786636395] req@00000000f6d1affb x1873423893578112/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786636411 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8329.715360] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8329.753460] Lustre: 254827:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 8329.793134] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8330.246991] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8339.407713] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8380.907354] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8380.914737] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8380.915527] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8380.925068] Lustre: Skipped 1 previous similar message [ 8386.183533] Lustre: server umount lustre-OST0000 complete [ 8391.138349] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8391.158260] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8391.162329] Lustre: Skipped 1 previous similar message [ 8391.178345] Lustre: Skipped 2 previous similar messages [ 8396.257936] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8396.269817] Lustre: Skipped 1 previous similar message [ 8396.939807] Lustre: server umount lustre-MDT0000 complete [ 8401.380846] LustreError: 259974:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8401.400387] LustreError: 259974:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 8401.647862] LustreError: 255497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786636483 with bad export cookie 5629418161124465606 [ 8401.657524] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8401.658784] LustreError: 255497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8402.032161] Lustre: server umount lustre-MDT0001 complete [ 8414.355545] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 8417.891547] Key type lgssc unregistered [ 8418.389683] LNet: 260930:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8418.403392] LNetError: 260930:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8419.429815] LNet: Removed LNI 192.168.202.132@tcp [ 8420.288981] Key type .llcrypt unregistered [ 8420.292151] Key type ._llcrypt unregistered [ 8446.239328] Key type ._llcrypt registered [ 8446.240647] Key type .llcrypt registered [ 8446.375986] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 8448.273458] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8448.290439] alg: No test for adler32 (adler32-zlib) [ 8449.470415] Lustre: Lustre: Build Version: 2.17.57_2_g7a6d6aa [ 8449.897423] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 8451.560470] Key type lgssc registered [ 8453.084681] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8461.494688] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8461.520044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8462.967229] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8467.997374] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8472.550921] LustreError: 262352:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8472.583944] LustreError: 262352:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8477.220509] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8477.554801] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8482.230795] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8491.029745] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8497.401806] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8504.612556] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8504.930916] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8508.026585] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8511.239957] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8519.577897] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8548.320968] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8548.329547] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8548.334042] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8548.350601] Lustre: Skipped 1 previous similar message [ 8548.366544] Lustre: Skipped 1 previous similar message [ 8552.746282] Lustre: server umount lustre-OST0000 complete [ 8558.563623] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8558.567712] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8558.610479] Lustre: Skipped 1 previous similar message [ 8558.640874] Lustre: Skipped 1 previous similar message [ 8563.701378] LustreError: 263079:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8563.718232] LustreError: 263079:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8564.025584] Lustre: server umount lustre-MDT0000 complete [ 8568.811512] LustreError: 262352:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8568.842071] LustreError: 262352:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8568.910829] LustreError: 262330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786636650 with bad export cookie 16505543141563887614 [ 8568.919290] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8569.379238] Lustre: server umount lustre-MDT0001 complete [ 8582.323351] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 8585.353339] Key type lgssc unregistered [ 8585.643761] LNet: 266661:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8585.650747] LNetError: 266661:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8585.689351] LNet: Removed LNI 192.168.202.132@tcp [ 8586.642863] Key type .llcrypt unregistered [ 8586.644990] Key type ._llcrypt unregistered [ 8606.581413] Key type ._llcrypt registered [ 8606.589705] Key type .llcrypt registered [ 8606.721933] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 8608.417703] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8608.482076] alg: No test for adler32 (adler32-zlib) [ 8609.606535] Lustre: Lustre: Build Version: 2.17.57_2_g7a6d6aa [ 8609.932085] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 8611.632037] Key type lgssc registered [ 8613.298675] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8622.485363] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8622.499254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8623.860394] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8628.679381] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8633.321085] LustreError: 267886:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8633.343445] LustreError: 267886:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8637.619357] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8637.937359] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8642.566990] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8650.447164] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8656.469585] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8664.318706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8664.744498] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8668.859144] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8670.737042] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8679.747912] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8692.704436] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8692.708949] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8692.722106] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8692.751056] Lustre: Skipped 1 previous similar message [ 8697.825128] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8697.829303] Lustre: Skipped 1 previous similar message [ 8698.923043] Lustre: server umount lustre-OST0000 complete [ 8702.944442] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8702.961470] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8702.972231] Lustre: Skipped 1 previous similar message [ 8708.071661] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8708.075179] Lustre: Skipped 1 previous similar message [ 8708.874696] Lustre: server umount lustre-MDT0000 complete [ 8712.205450] LustreError: 267865:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786636794 with bad export cookie 16586714939569757646 [ 8712.208728] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8712.213414] LustreError: 267865:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8712.537157] Lustre: server umount lustre-MDT0001 complete [ 8723.554449] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 8726.421767] Key type lgssc unregistered [ 8726.683765] LNet: 271565:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8726.693988] LNetError: 271565:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8726.726445] LNet: Removed LNI 192.168.202.132@tcp [ 8727.717405] Key type .llcrypt unregistered [ 8727.720783] Key type ._llcrypt unregistered [ 8749.608802] Key type ._llcrypt registered [ 8749.614246] Key type .llcrypt registered [ 8749.723787] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 8751.346221] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8751.381083] alg: No test for adler32 (adler32-zlib) [ 8752.440130] Lustre: Lustre: Build Version: 2.17.57_2_g7a6d6aa [ 8752.698540] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 8754.393520] Key type lgssc registered [ 8755.510832] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8762.501790] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8762.518655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8763.932669] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8768.449225] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8773.089750] LustreError: 272981:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8773.113165] LustreError: 272981:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8777.648800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8778.078184] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8783.653123] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8791.860469] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8797.933934] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8805.236355] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8805.528872] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8808.623413] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8811.713572] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8820.128711] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8843.746184] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8843.753048] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8843.758944] Lustre: Skipped 1 previous similar message [ 8848.869911] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8848.874450] Lustre: Skipped 1 previous similar message [ 8849.134593] Lustre: server umount lustre-OST0000 complete [ 8853.989937] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8854.001978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8854.015903] Lustre: Skipped 1 previous similar message [ 8854.027224] Lustre: Skipped 2 previous similar messages [ 8858.925982] Lustre: server umount lustre-MDT0000 complete [ 8859.105109] LustreError: 272982:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8859.116031] LustreError: 272982:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8862.915660] LustreError: 272961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786636944 with bad export cookie 7294198115952022513 [ 8862.921070] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8863.242376] Lustre: server umount lustre-MDT0001 complete [ 8874.752643] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 8877.700152] Key type lgssc unregistered [ 8878.065481] LNet: 277299:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8878.074415] LNetError: 277299:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8879.141856] LNet: Removed LNI 192.168.202.132@tcp [ 8879.953670] Key type .llcrypt unregistered [ 8879.957723] Key type ._llcrypt unregistered [ 8898.829747] Key type ._llcrypt registered [ 8898.831181] Key type .llcrypt registered [ 8898.935756] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 8900.111487] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8900.163877] alg: No test for adler32 (adler32-zlib) [ 8901.318809] Lustre: Lustre: Build Version: 2.17.57_2_g7a6d6aa [ 8901.542272] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 8903.241666] Key type lgssc registered [ 8904.790311] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8912.956603] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8913.026342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8914.478292] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8919.514761] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8924.670537] LustreError: 278535:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8924.694047] LustreError: 278535:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8928.980960] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8929.378135] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8934.675715] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8942.884884] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8949.032418] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8957.321452] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8957.696394] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8959.769738] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8964.317974] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8972.016557] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8989.667104] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8989.672990] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8989.706246] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8989.726928] Lustre: Skipped 1 previous similar message [ 8993.502639] Lustre: server umount lustre-OST0000 complete [ 8999.906047] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8999.921109] Lustre: Skipped 1 previous similar message [ 8999.925937] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9005.024583] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9005.032304] Lustre: Skipped 2 previous similar messages [ 9009.122097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9009.129394] Lustre: Skipped 1 previous similar message [ 9012.191114] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9012.614874] Lustre: server umount lustre-MDT0000 complete [ 9014.241162] LustreError: 278548:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9014.261956] LustreError: 278548:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9016.738875] LustreError: 278501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786637098 with bad export cookie 5078300506009565463 [ 9016.762401] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9017.160748] Lustre: server umount lustre-MDT0001 complete [ 9029.912324] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 9033.053485] Key type lgssc unregistered [ 9033.393433] LNet: 282241:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9033.404654] LNetError: 282241:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9033.429960] LNet: Removed LNI 192.168.202.132@tcp [ 9034.495317] Key type .llcrypt unregistered [ 9034.497087] Key type ._llcrypt unregistered [ 9047.854522] Key type ._llcrypt registered [ 9047.856745] Key type .llcrypt registered [ 9048.042035] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 9066.171187] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [ 9083.359829] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 9084.746922] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9084.771490] alg: No test for adler32 (adler32-zlib) [ 9085.872085] Lustre: Lustre: Build Version: 2.17.57_2_g7a6d6aa [ 9086.112703] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 9087.807208] Key type lgssc registered [ 9089.035398] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9130.395682] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9130.426126] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9131.874289] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9131.951883] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9132.068768] Lustre: lustre-MDT0000: new disk, initializing [ 9132.151073] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9132.171414] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9136.430724] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 9146.618750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9146.747097] Lustre: 285795:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9146.821548] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9146.829059] Lustre: Skipped 1 previous similar message [ 9146.941618] Lustre: lustre-MDT0001: new disk, initializing [ 9147.077032] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9147.105598] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9147.113668] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9151.957764] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 9162.900590] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9169.561360] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9176.430986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9176.805873] Lustre: lustre-OST0000: new disk, initializing [ 9176.809480] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9176.814886] Lustre: 287764:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9176.939676] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9184.267115] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 9184.355413] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9184.369157] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9184.514338] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9194.136203] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9198.600479] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9198.768476] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9202.490861] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9202.638775] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9205.224058] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9205.226286] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9205.229285] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9205.240912] LustreError: Skipped 1 previous similar message [ 9205.248103] Lustre: Skipped 1 previous similar message [ 9210.338761] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9210.343099] Lustre: Skipped 1 previous similar message [ 9210.470800] Lustre: server umount lustre-OST0000 complete [ 9215.458750] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9215.462038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9215.472486] Lustre: Skipped 1 previous similar message [ 9220.577530] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9220.595147] Lustre: Skipped 2 previous similar messages [ 9220.940222] Lustre: server umount lustre-MDT0000 complete [ 9226.306601] LustreError: 285788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786637308 with bad export cookie 10773387500260998382 [ 9226.321707] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9226.898704] Lustre: server umount lustre-MDT0001 complete [ 9237.638321] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 12:08:36 (1786637316) [ 9246.014099] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9246.718372] LustreError: 290179:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9246.734657] LustreError: 290179:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9246.796308] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9251.810912] LustreError: 290180:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9252.349346] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 9256.935175] LustreError: 290179:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9260.447737] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9260.766206] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9265.359878] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 9273.489397] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9278.762633] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9285.662631] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9286.092492] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9292.839849] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 9301.113422] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9315.820781] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9315.825832] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9315.832504] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9315.841476] Lustre: Skipped 1 previous similar message [ 9315.859508] Lustre: Skipped 1 previous similar message [ 9316.568263] Lustre: server umount lustre-OST0000 complete [ 9320.602916] Lustre: server umount lustre-MDT0000 complete [ 9320.942420] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9320.944669] LustreError: 290176:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9320.961055] Lustre: Skipped 1 previous similar message [ 9320.993030] LustreError: 290176:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9324.396939] LustreError: 290161:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786637406 with bad export cookie 10773387500260999264 [ 9324.407659] LustreError: 290161:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9324.410831] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9326.881191] Lustre: server umount lustre-MDT0001 complete [ 9338.351433] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 9341.044763] Key type lgssc unregistered [ 9341.298475] LNet: 293737:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9341.307481] LNetError: 293737:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9342.373741] LNet: Removed LNI 192.168.202.132@tcp [ 9343.226447] Key type .llcrypt unregistered [ 9343.231387] Key type ._llcrypt unregistered [ 9370.535971] Key type ._llcrypt registered [ 9370.542360] Key type .llcrypt registered [ 9370.683362] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [ 9387.758829] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 9389.386466] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9389.429161] alg: No test for adler32 (adler32-zlib) [ 9390.531485] Lustre: Lustre: Build Version: 2.17.57_2_g7a6d6aa [ 9390.826293] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 9392.551323] Key type lgssc registered [ 9394.124565] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9442.619882] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9442.647719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9444.007296] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9444.038227] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9444.113727] Lustre: lustre-MDT0000: new disk, initializing [ 9444.284323] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9444.303908] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9448.865791] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 9458.946175] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9459.081856] Lustre: 297564:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9459.154490] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9459.157368] Lustre: Skipped 1 previous similar message [ 9459.226887] Lustre: lustre-MDT0001: new disk, initializing [ 9459.313077] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9459.351077] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9459.364718] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9464.868555] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 9475.853747] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9481.824540] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9488.913464] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9489.217844] Lustre: lustre-OST2710: new disk, initializing [ 9489.220373] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9489.226671] Lustre: 299534:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9489.303257] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9494.660920] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9494.674432] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9494.816245] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9495.119288] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 9504.981315] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9511.281173] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9511.450289] Lustre: lustre-OST03e8: new disk, initializing [ 9511.453115] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9511.463925] Lustre: 300599:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9511.586155] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9517.227381] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 9520.823704] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9520.835079] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9520.995122] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9526.224855] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9535.097786] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9535.397260] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9540.505299] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9540.770228] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9545.406606] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9545.624432] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9550.027808] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9550.218811] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9556.451439] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9556.469137] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9556.484026] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9556.974510] Lustre: lustre-MDT0000-lwp-OST03e8: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9556.987016] Lustre: Skipped 2 previous similar messages [ 9556.996299] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9557.000792] Lustre: Skipped 2 previous similar messages [ 9560.288246] Lustre: server umount lustre-MDT0000 complete [ 9564.414202] LustreError: 297557:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786637646 with bad export cookie 6711571675407155027 [ 9564.416445] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9564.423795] LustreError: 297557:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9564.982708] Lustre: server umount lustre-MDT0001 complete [ 9579.473585] Lustre: server umount lustre-OST2710 complete [ 9583.586300] Lustre: 295365:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786637649/real 1786637649] req@00000000c766a719 x1873425173907584/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786637665 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9583.625805] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9584.692206] Lustre: server umount lustre-OST03e8 complete [ 9605.721385] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [ 9616.429921] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 9654.708828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9654.975196] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9655.002806] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9655.084745] Lustre: lustre-MDT0000: new disk, initializing [ 9655.161896] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9655.181061] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9660.647807] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 9672.113247] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9672.239588] Lustre: 305849:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9672.285799] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9672.288523] Lustre: Skipped 1 previous similar message [ 9672.361788] Lustre: lustre-MDT0001: new disk, initializing [ 9672.520469] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9672.559186] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9672.566198] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9677.448695] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 9686.643260] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9692.505243] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9700.685530] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9701.041897] Lustre: lustre-OST0000: new disk, initializing [ 9701.047386] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9701.054767] Lustre: 307822:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9701.203433] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9708.159509] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 9710.174965] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9710.190593] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9710.321314] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9718.218080] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9723.158227] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9723.398303] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9727.736422] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9727.979216] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9731.053105] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9731.055384] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9731.059208] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9731.063188] LustreError: Skipped 1 previous similar message [ 9731.091770] Lustre: Skipped 1 previous similar message [ 9736.177433] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9736.185356] Lustre: Skipped 2 previous similar messages [ 9741.289165] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9741.303993] Lustre: Skipped 1 previous similar message [ 9744.351564] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9744.527585] Lustre: server umount lustre-OST0000 complete [ 9749.473149] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9749.478104] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9749.489161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9755.106234] Lustre: server umount lustre-MDT0000 complete [ 9756.649949] LustreError: 305863:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9756.674554] LustreError: 305863:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 9758.986878] LustreError: 306646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786637840 with bad export cookie 6711571675407156707 [ 9758.991621] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9758.995973] LustreError: 306646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9759.283722] Lustre: server umount lustre-MDT0001 complete [ 9767.918261] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 12:17:27 (1786637847) [ 9769.418495] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9771.240442] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 12:17:30 (1786637850) [ 9774.613408] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 9800.940980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9801.284759] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9801.310793] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9801.517099] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9806.418281] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 9815.908101] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9816.014150] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9816.059966] Lustre: 311677:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9821.613518] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 9845.297192] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9845.529922] Lustre: Denying initial registration attempt from nid 192.168.202.132@tcp, specified as failover [ 9845.534596] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9845.539612] LustreError: 313130:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 9845.542818] LustreError: 313130:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9845.546269] LustreError: 313130:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9845.578732] Lustre: server umount lustre-OST0000 complete [ 9845.581687] LustreError: 313130:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9852.391084] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9852.402904] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9852.418822] Lustre: Skipped 1 previous similar message [ 9852.427246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9852.440052] Lustre: Skipped 2 previous similar messages [ 9860.231824] Lustre: server umount lustre-MDT0000 complete [ 9860.580071] LustreError: 312803:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9860.599470] LustreError: 312803:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9864.089745] LustreError: 311669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786637946 with bad export cookie 6711571675407157589 [ 9864.092963] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9864.107503] LustreError: 311669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9864.451714] Lustre: server umount lustre-MDT0001 complete [ 9876.585511] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 9879.795219] Key type lgssc unregistered [ 9880.106690] LNet: 314255:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9880.118058] LNetError: 314255:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9880.136641] LNet: Removed LNI 192.168.202.132@tcp [ 9881.133277] Key type .llcrypt unregistered [ 9881.134988] Key type ._llcrypt unregistered [ 9893.161579] Key type ._llcrypt registered [ 9893.164114] Key type .llcrypt registered [ 9893.279352] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 9894.763582] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9894.791452] alg: No test for adler32 (adler32-zlib) [ 9895.966842] Lustre: Lustre: Build Version: 2.17.57_2_g7a6d6aa [ 9896.281735] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 9898.001610] Key type lgssc registered [ 9899.323727] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9952.510953] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 9963.466533] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9963.493148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9964.944959] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9964.971966] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9965.125865] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9970.590105] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 9980.298651] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9980.362146] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9980.387863] Lustre: 317022:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9980.654537] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9985.692741] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 9994.976443] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10002.007969] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10023.591973] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10023.907637] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10024.110497] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10030.377783] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [10037.968938] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10044.900665] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10044.901808] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10044.913090] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10044.921029] LustreError: Skipped 1 previous similar message [10046.264837] Lustre: server umount lustre-OST0000 complete [10050.016907] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10050.023989] Lustre: Skipped 1 previous similar message [10050.027935] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10050.034177] Lustre: Skipped 2 previous similar messages [10055.136573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10055.140801] Lustre: Skipped 1 previous similar message [10060.259434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10060.266842] Lustre: Skipped 1 previous similar message [10064.351964] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10064.588238] Lustre: server umount lustre-MDT0000 complete [10068.416523] LustreError: 317013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786638150 with bad export cookie 13104008387326951563 [10068.419796] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10068.434089] LustreError: 317013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10068.817798] Lustre: server umount lustre-MDT0001 complete [10081.228111] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [10084.057549] Key type lgssc unregistered [10084.318922] LNet: 320597:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10084.323118] LNetError: 320597:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10084.332704] LNet: Removed LNI 192.168.202.132@tcp [10084.999401] Key type .llcrypt unregistered [10085.001588] Key type ._llcrypt unregistered [10114.082363] Key type ._llcrypt registered [10114.087472] Key type .llcrypt registered [10114.225101] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [10115.468816] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10115.621476] alg: No test for adler32 (adler32-zlib) [10116.904616] Lustre: Lustre: Build Version: 2.17.57_2_g7a6d6aa [10117.284301] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [10119.024914] Key type lgssc registered [10120.168961] Lustre: Echo OBD driver; http://www.lustre.org/ [10127.446639] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10127.467507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10129.094889] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10134.095085] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [10139.625211] LustreError: 322014:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10139.654348] LustreError: 322014:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10143.385233] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10143.792635] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10149.111832] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [10156.819724] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10162.055507] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10170.421093] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10170.867724] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10177.360947] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [10184.350547] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10200.034983] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10200.047224] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10200.070420] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10201.614605] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.32@tcp (stopping) [10203.618732] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10203.620881] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10203.636357] Lustre: Skipped 2 previous similar messages [10206.690392] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.32@tcp (stopping) [10211.811367] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.32@tcp (stopping) [10211.826529] Lustre: Skipped 3 previous similar messages [10213.343925] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10213.777691] Lustre: server umount lustre-MDT0000 complete [10213.858722] LustreError: 322009:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10213.870186] LustreError: 322009:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10216.944608] LustreError: 323445:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10217.624031] LustreError: 321995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786638299 with bad export cookie 2300912763946079873 [10217.632654] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10217.636591] LustreError: 321995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10218.977490] LustreError: 323445:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10218.977923] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10219.002082] Lustre: Skipped 1 previous similar message [10222.079692] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.32@tcp (stopping) [10222.088536] Lustre: Skipped 1 previous similar message [10223.008686] LustreError: 322009:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10223.025974] LustreError: 322009:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10224.112524] Lustre: server umount lustre-MDT0001 complete [10232.892910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10245.317894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10263.012219] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x1fee7bf7d22b3528 [10263.031944] Lustre: MGC192.168.202.132@tcp: Connection restored to 0@lo (at 0@lo) [10263.463355] LustreError: 323847:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10263.595724] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10263.681589] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [10268.988826] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [10277.865896] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10278.616572] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10279.105484] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10284.514634] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10284.863643] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [10294.867346] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10296.881570] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10304.955206] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10307.190911] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10319.843071] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10319.843619] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10319.849844] LustreError: Skipped 1 previous similar message [10319.850550] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10319.866099] Lustre: Skipped 1 previous similar message [10319.888374] Lustre: Skipped 2 previous similar messages [10322.314964] Lustre: server umount lustre-OST0000 complete [10330.080592] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10330.088368] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10333.074883] Lustre: server umount lustre-MDT0000 complete [10335.714371] LustreError: 327561:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10335.723915] LustreError: 327561:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [10337.522322] LustreError: 321994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786638419 with bad export cookie 2300912763946087720 [10337.536612] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10337.540434] LustreError: 321994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10337.882380] Lustre: server umount lustre-MDT0001 complete [10350.757585] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [10354.387912] Key type lgssc unregistered [10354.851941] LNet: 328316:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10354.861824] LNetError: 328316:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10355.882660] LNet: Removed LNI 192.168.202.132@tcp [10357.380611] Key type .llcrypt unregistered [10357.397242] Key type ._llcrypt unregistered [10386.429134] Key type ._llcrypt registered [10386.432493] Key type .llcrypt registered [10386.543862] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [10387.723559] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10387.759119] alg: No test for adler32 (adler32-zlib) [10388.867434] Lustre: Lustre: Build Version: 2.17.57_2_g7a6d6aa [10389.095324] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [10390.839178] Key type lgssc registered [10391.677595] Lustre: Echo OBD driver; http://www.lustre.org/ [10398.665494] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10398.691960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10400.284734] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10404.673856] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [10410.466506] LustreError: 329733:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10410.497774] LustreError: 329733:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10413.329540] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10413.854554] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10418.253232] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [10426.853699] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10433.466686] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10437.360466] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10437.384905] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10437.397491] Lustre: Skipped 1 previous similar message [10438.434392] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10443.775814] LustreError: 329733:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10443.798634] LustreError: 329733:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10443.985459] Lustre: server umount lustre-MDT0000 complete [10448.607884] LustreError: 329715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786638530 with bad export cookie 17123920593672665193 [10448.630489] LustreError: 329715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10448.636335] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10449.134160] Lustre: server umount lustre-MDT0001 complete [10465.951400] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10466.401198] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10466.448860] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10466.730846] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10472.064841] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [10482.562689] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10482.773656] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10482.837371] Lustre: 332321:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10483.436485] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10488.828596] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [10497.699159] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10505.223333] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10516.410690] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10516.724586] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10516.754574] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10516.767563] LustreError: 334206:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [10516.776851] LustreError: 334206:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [10516.788612] LustreError: 334206:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10516.959325] Lustre: server umount lustre-OST0000 complete [10516.982077] LustreError: 334206:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [10528.302974] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10528.590969] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10528.871360] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10534.028961] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10535.750789] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [10543.084791] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10551.108778] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10551.156033] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10551.232498] Lustre: lustre-OST0001: new disk, initializing [10551.235895] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10551.244387] Lustre: 335751:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10551.342633] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10558.839719] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [10559.153676] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10559.164098] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10559.342443] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10567.715265] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10574.832963] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10574.839150] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10574.847861] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10574.847870] Lustre: Skipped 1 previous similar message [10574.886467] Lustre: Skipped 2 previous similar messages [10577.500060] Lustre: server umount lustre-OST0001 complete [10579.953532] LustreError: 334984:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10579.989157] LustreError: 334984:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10585.058070] LustreError: 334674:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10585.059414] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10585.060392] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10585.060398] LustreError: Skipped 1 previous similar message [10585.062028] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10585.062035] Lustre: Skipped 1 previous similar message [10585.094175] LustreError: 334674:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10585.137722] Lustre: Skipped 1 previous similar message [10588.950073] Lustre: server umount lustre-OST0000 complete [10595.298968] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10595.299681] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10595.311442] Lustre: Skipped 1 previous similar message [10595.314391] Lustre: Skipped 1 previous similar message [10600.416952] LustreError: 332332:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10602.388573] Lustre: server umount lustre-MDT0000 complete [10606.328047] LustreError: 332313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786638688 with bad export cookie 17123920593672665746 [10606.340450] LustreError: 332313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10606.344300] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10606.960338] Lustre: server umount lustre-MDT0001 complete [10619.579901] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [10623.973181] Key type lgssc unregistered [10624.353691] LNet: 337716:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10624.362094] LNetError: 337716:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10624.381724] LNet: Removed LNI 192.168.202.132@tcp [10625.180479] Key type .llcrypt unregistered [10625.181987] Key type ._llcrypt unregistered [10704.550781] Key type ._llcrypt registered [10704.553260] Key type .llcrypt registered [10704.695802] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [10725.465935] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [10727.951954] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10728.094525] alg: No test for adler32 (adler32-zlib) [10729.340568] Lustre: Lustre: Build Version: 2.17.57_2_g7a6d6aa [10729.551035] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [10731.165285] Key type lgssc registered [10733.142523] Lustre: Echo OBD driver; http://www.lustre.org/ [10773.577795] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10773.627882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10774.933235] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10774.997987] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10775.108872] Lustre: lustre-MDT0000: new disk, initializing [10775.206700] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10775.237227] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10779.865471] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [10790.971855] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10791.112715] Lustre: 342655:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10791.160193] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10791.164139] Lustre: Skipped 1 previous similar message [10791.320856] Lustre: lustre-MDT0001: new disk, initializing [10791.519464] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10791.543992] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10791.557452] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10796.062420] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [10806.247321] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10812.659478] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10821.874313] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10822.463474] Lustre: lustre-OST0000: new disk, initializing [10822.467945] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10822.476532] Lustre: 344626:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10822.643740] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10831.020384] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10831.036677] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10831.340022] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10832.227578] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [10842.866690] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10849.057856] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10849.398448] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10854.364200] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10854.571919] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10856.928770] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10856.938995] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10856.941235] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10856.985298] Lustre: Skipped 1 previous similar message [10862.049679] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10862.061834] Lustre: Skipped 2 previous similar messages [10862.809407] Lustre: server umount lustre-OST0000 complete [10867.174320] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10867.190058] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10867.198426] Lustre: Skipped 1 previous similar message [10867.210296] Lustre: Skipped 1 previous similar message [10872.290159] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10872.299904] Lustre: Skipped 1 previous similar message [10872.884986] Lustre: server umount lustre-MDT0000 complete [10876.883219] LustreError: 343459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786638958 with bad export cookie 11509250292724401796 [10876.885912] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10876.894096] LustreError: 343459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10877.297315] Lustre: server umount lustre-MDT0001 complete [10886.864091] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 12:36:06 (1786638966) [10895.325961] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 12:36:15 (1786638975) [10903.660583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10904.212535] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10908.604753] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [10909.668508] LustreError: 347382:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10909.695823] LustreError: 347382:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10914.785679] LustreError: 347383:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10917.153709] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10917.478631] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10922.114334] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [10930.354522] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10937.136960] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10945.998732] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10946.442608] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10954.487884] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [10961.725158] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10969.417600] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10971.554908] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10973.210884] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10975.236824] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10977.516987] Lustre: Failing over lustre-MDT0000 [10977.897250] Lustre: server umount lustre-MDT0000 complete [10978.279716] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10978.287324] LustreError: 347379:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10978.289446] Lustre: Skipped 1 previous similar message [10983.397540] LustreError: 347382:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10983.427167] LustreError: 347382:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [10988.514663] LustreError: 348622:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10988.535095] LustreError: 348622:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10994.655111] Lustre: 340878:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786639060/real 1786639060] req@000000001faba070 x1873426577956352/t0(0) o400->MGC192.168.202.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786639076 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10994.686704] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10998.215517] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10999.274211] LustreError: 347377:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10999.286343] LustreError: 347377:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [11004.897562] LustreError: 340877:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000689f6e9e x1873426577961600/t0(0) o250->MGC192.168.202.132@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11005.239825] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11005.301596] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11009.507942] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11010.551465] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11010.593174] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [11010.646454] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11011.039053] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [11021.004323] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11022.771687] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11026.706842] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [11029.083624] Lustre: Failing over lustre-MDT0000 [11029.708395] Lustre: server umount lustre-MDT0000 complete [11030.019109] LustreError: 347379:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11030.038663] LustreError: 347379:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [11031.025163] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11031.030390] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11031.048644] Lustre: Skipped 1 previous similar message [11047.776199] Lustre: 340881:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786639113/real 1786639113] req@00000000e5a5dbbd x1873426577978496/t0(0) o400->MGC192.168.202.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786639129 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11047.803271] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11049.888231] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11057.376580] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11057.433880] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11061.735772] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11062.129313] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [11062.769996] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [11062.792414] Lustre: Skipped 2 previous similar messages [11062.838022] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [11062.892314] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11073.729092] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11075.448525] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11083.233241] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11083.240570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11083.265887] Lustre: Skipped 3 previous similar messages [11083.294652] Lustre: Skipped 2 previous similar messages [11087.725402] Lustre: server umount lustre-MDT0000 complete [11088.353573] LustreError: 348622:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11088.372163] LustreError: 348622:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 29 previous similar messages [11092.145544] LustreError: 350153:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786639174 with bad export cookie 11509250292724404519 [11092.152984] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11092.155885] LustreError: 350153:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11092.553487] Lustre: server umount lustre-MDT0001 complete [11107.199957] Lustre: server umount lustre-OST0000 complete [11123.881766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11124.456688] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11129.468268] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [11140.487668] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11147.816133] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [11157.665554] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11165.445768] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11173.892357] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11181.264916] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [11189.077452] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11194.873403] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11206.129637] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11206.141057] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11206.162865] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11210.226203] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11211.394914] Lustre: server umount lustre-OST0000 complete [11223.374053] Lustre: server umount lustre-MDT0000 complete [11225.570532] LustreError: 353722:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11225.576854] LustreError: 353722:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [11227.718437] LustreError: 353703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786639309 with bad export cookie 11509250292724405261 [11227.719898] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11227.724367] LustreError: 353703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11228.018760] Lustre: server umount lustre-MDT0001 complete [11238.731323] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [11241.820971] Key type lgssc unregistered [11242.163303] LNet: 357287:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11242.176347] LNetError: 357287:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11242.208585] LNet: Removed LNI 192.168.202.132@tcp [11243.386361] Key type .llcrypt unregistered [11243.388959] Key type ._llcrypt unregistered [11273.702472] Key type ._llcrypt registered [11273.704901] Key type .llcrypt registered [11273.807348] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [11276.185590] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11276.214688] alg: No test for adler32 (adler32-zlib) [11277.608778] Lustre: Lustre: Build Version: 2.17.57_2_g7a6d6aa [11277.892108] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [11279.623328] Key type lgssc registered [11281.762626] Lustre: Echo OBD driver; http://www.lustre.org/ [11293.411636] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11293.431033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11294.816175] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11299.355658] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [11304.929865] LustreError: 358808:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11304.946607] LustreError: 358808:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11308.544181] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11308.930436] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11313.701649] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [11320.851822] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11326.907513] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11334.771068] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11335.299193] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11339.395213] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11342.181626] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [11350.565677] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11366.294546] LustreError: 361405:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x2:0x0].0x0 (000000003138f6c7) refcount nonzero (1) after lock cleanup; forcing cleanup. [11368.499829] Lustre: server umount lustre-MDT0000 complete [11369.444151] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11369.453581] LustreError: 358808:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11369.489107] Lustre: Skipped 2 previous similar messages [11369.545063] LustreError: 358808:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11373.895720] LustreError: 358789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786639455 with bad export cookie 18231385036308530376 [11373.904618] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11373.913480] LustreError: 358789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11374.561501] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11374.566074] LustreError: 360647:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11374.607890] LustreError: 360647:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11374.783053] Lustre: server umount lustre-MDT0001 complete [11380.559097] Lustre: server umount lustre-OST0000 complete [11394.326807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11394.610594] LustreError: 362536:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11394.679536] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11400.171601] LustreError: 362537:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11400.492662] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [11409.387441] LustreError: 362536:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11409.405626] LustreError: 362536:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11410.285326] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11410.723442] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11415.750550] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [11424.307970] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11430.369930] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11437.207214] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11437.567597] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11439.606938] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11444.681735] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [11453.298524] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11466.222287] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11466.229927] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11466.241622] Lustre: Skipped 1 previous similar message [11466.272214] Lustre: Skipped 2 previous similar messages [11471.169312] Lustre: server umount lustre-MDT0000 complete [11471.345502] LustreError: 362537:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11474.780777] LustreError: 363271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786639556 with bad export cookie 18231385036308531489 [11474.791928] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11474.795152] LustreError: 363271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11475.552379] Lustre: server umount lustre-MDT0001 complete [11489.213716] Lustre: server umount lustre-OST0000 complete [11504.337975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11504.792308] LustreError: 366446:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11504.821100] LustreError: 366446:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11504.871663] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11509.910316] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [11518.515392] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11524.605767] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [11533.020428] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11538.419857] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11546.299540] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11546.688840] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11546.693395] Lustre: Skipped 1 previous similar message [11549.797127] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11553.095784] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [11561.627061] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11568.609273] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11568.611753] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11568.618628] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11568.624501] Lustre: Skipped 2 previous similar messages [11573.153325] Lustre: server umount lustre-OST0000 complete [11578.859205] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11578.866262] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11578.889632] Lustre: Skipped 1 previous similar message [11578.898226] Lustre: Skipped 1 previous similar message [11583.970831] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11583.979880] Lustre: Skipped 1 previous similar message [11584.610667] Lustre: server umount lustre-MDT0000 complete [11588.073868] LustreError: 366443:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11588.087994] LustreError: 366443:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11588.636758] LustreError: 366428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786639670 with bad export cookie 18231385036308532511 [11588.654694] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11588.970507] Lustre: server umount lustre-MDT0001 complete [11599.987973] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [11603.413918] Key type lgssc unregistered [11603.744895] LNet: 369982:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11603.752306] LNetError: 369982:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11603.781277] LNet: Removed LNI 192.168.202.132@tcp [11604.641462] Key type .llcrypt unregistered [11604.643114] Key type ._llcrypt unregistered [11630.228863] Key type ._llcrypt registered [11630.236135] Key type .llcrypt registered [11630.459869] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [11631.696385] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11631.717040] alg: No test for adler32 (adler32-zlib) [11633.022854] Lustre: Lustre: Build Version: 2.17.57_2_g7a6d6aa [11633.311118] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [11635.031185] Key type lgssc registered [11636.375692] Lustre: Echo OBD driver; http://www.lustre.org/ [11644.396576] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11644.410933] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11644.421221] LustreError: 371424:0:(osd_handler.c:8553:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11644.436169] LustreError: 371424:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11644.440531] LustreError: 371424:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11644.446549] LustreError: 371424:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11644.457275] LustreError: 371424:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11652.678937] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11652.688185] LustreError: 371822:0:(osd_handler.c:8553:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11652.704303] LustreError: 371822:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11652.709550] LustreError: 371822:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11652.714699] LustreError: 371822:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11652.722665] LustreError: 371822:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11673.990770] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [11676.626794] Key type lgssc unregistered [11676.965510] LNet: 372828:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11676.970882] LNetError: 372828:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11676.993074] LNet: Removed LNI 192.168.202.132@tcp [11677.819449] Key type .llcrypt unregistered [11677.825927] Key type ._llcrypt unregistered [11699.334978] Key type ._llcrypt registered [11699.339402] Key type .llcrypt registered [11699.416596] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [11715.814353] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [11716.940118] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11717.031451] alg: No test for adler32 (adler32-zlib) [11718.117347] Lustre: Lustre: Build Version: 2.17.57_2_g7a6d6aa [11718.359564] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [11720.071259] Key type lgssc registered [11721.531344] Lustre: Echo OBD driver; http://www.lustre.org/ [11761.103921] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11761.116526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11762.385898] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11762.438381] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11762.597717] Lustre: lustre-MDT0000: new disk, initializing [11762.719282] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11762.750401] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11767.889590] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [11779.050064] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11779.181988] Lustre: 375987:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11779.227662] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11779.232506] Lustre: Skipped 1 previous similar message [11779.306418] Lustre: lustre-MDT0001: new disk, initializing [11779.400808] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11779.436548] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11779.452962] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11786.247466] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [11800.150957] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11808.093579] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11816.279670] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11816.747577] Lustre: lustre-OST0000: new disk, initializing [11816.753356] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11816.757729] Lustre: 377958:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11816.805425] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11824.019571] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [11826.318310] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11826.338792] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11826.520938] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11835.768434] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11841.214440] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11841.481755] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11846.280675] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11846.476229] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11852.269791] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11852.270052] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11852.278971] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11852.321800] Lustre: Skipped 1 previous similar message [11854.637161] Lustre: server umount lustre-OST0000 complete [11861.485532] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11861.491622] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11861.533080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11861.540641] Lustre: Skipped 1 previous similar message [11862.500371] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11865.037437] Lustre: server umount lustre-MDT0000 complete [11869.162687] LustreError: 376781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786639951 with bad export cookie 13145156650426810608 [11869.171603] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11869.186086] LustreError: 376781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11869.777462] Lustre: server umount lustre-MDT0001 complete [11881.838855] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11883.857599] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 12:52:43 (1786639963) [11892.498360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11893.078870] LustreError: 380417:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11893.106552] LustreError: 380417:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11893.145603] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11897.941498] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [11898.337194] LustreError: 380418:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11903.457415] LustreError: 380417:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11907.018627] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11907.476226] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11912.399562] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [11920.621267] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11927.062420] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11934.294155] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11934.752091] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11940.985367] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [11950.051372] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11957.218920] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11957.421447] Lustre: lustre-OST0001: new disk, initializing [11957.425323] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11957.435341] Lustre: 383256:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11957.595086] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11963.578489] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [11967.098023] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11967.119190] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11967.245820] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11972.819405] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11977.696164] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11977.700049] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11977.706264] LustreError: Skipped 1 previous similar message [11977.712501] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11977.712510] Lustre: Skipped 2 previous similar messages [11977.724485] Lustre: Skipped 1 previous similar message [11980.802873] Lustre: lustre-OST0001: Not available for connect from 192.168.202.32@tcp (stopping) [11980.810879] Lustre: Skipped 1 previous similar message [11983.566396] Lustre: server umount lustre-OST0001 complete [11985.886372] LustreError: 382459:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11993.074488] LustreError: 382239:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11993.106198] LustreError: 382239:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12000.224089] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12000.231851] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12000.246588] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12000.249041] Lustre: Skipped 2 previous similar messages [12003.305643] LustreError: 382240:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12003.325229] LustreError: 382240:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12006.552590] Lustre: server umount lustre-OST0000 complete [12013.536228] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12013.552264] Lustre: Skipped 1 previous similar message [12013.562509] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12013.574258] Lustre: Skipped 3 previous similar messages [12016.711714] Lustre: server umount lustre-MDT0000 complete [12020.817623] LustreError: 380399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786640102 with bad export cookie 13145156650426811490 [12020.818993] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12020.828607] LustreError: 380399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12027.476376] Lustre: server umount lustre-MDT0001 complete [12038.730483] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [12041.806953] Key type lgssc unregistered [12042.136241] LNet: 385249:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12042.142754] LNetError: 385249:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12042.160111] LNet: Removed LNI 192.168.202.132@tcp [12042.833120] Key type .llcrypt unregistered [12042.836143] Key type ._llcrypt unregistered [12101.424711] Key type ._llcrypt registered [12101.427466] Key type .llcrypt registered [12101.571908] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [12103.182117] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12103.211081] alg: No test for adler32 (adler32-zlib) [12104.386518] Lustre: Lustre: Build Version: 2.17.57_2_g7a6d6aa [12104.705482] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [12106.451647] Key type lgssc registered [12107.614870] Lustre: Echo OBD driver; http://www.lustre.org/ [12116.268807] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12116.283601] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12117.604750] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12117.646735] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12117.809238] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12122.625554] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [12131.325050] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12131.397700] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12131.423884] Lustre: 388049:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12131.602318] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12136.980572] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [12144.826501] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12151.623661] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12157.419892] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12157.431282] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12157.442687] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12158.434124] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12158.437965] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12163.554825] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12163.558601] Lustre: Skipped 2 previous similar messages [12168.159176] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12168.706411] Lustre: server umount lustre-MDT0000 complete [12172.774781] LustreError: 388039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786640254 with bad export cookie 1283267776591423993 [12172.791730] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12172.935305] Lustre: server umount lustre-MDT0001 complete [12184.481982] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 12:57:44 (1786640264) [12191.127519] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12191.526514] LustreError: 390562:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12191.546019] LustreError: 390562:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12191.605248] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12195.845230] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [12196.832862] LustreError: 390563:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12201.954174] LustreError: 390562:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12205.312307] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12205.910557] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12211.339693] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [12220.602638] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12227.662385] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12235.065448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12235.261467] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [12235.491949] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12242.535601] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [12250.100918] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12258.936285] Lustre: Permanently deactivating lustre-OST0000 [12258.959914] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [12266.008726] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12267.449304] LustreError: 393502:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [12267.458814] LustreError: 393502:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12267.469727] LustreError: 393502:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12270.472845] LustreError: 393551:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12270.478239] LustreError: 393551:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12279.152795] Lustre: server umount lustre-OST0000 complete [12282.389257] LustreError: 393801:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12282.395604] LustreError: 393801:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12287.971498] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12287.984574] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12287.996511] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12294.628503] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12294.639953] Lustre: Skipped 3 previous similar messages [12294.758606] Lustre: server umount lustre-MDT0000 complete [12299.390161] LustreError: 390544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786640381 with bad export cookie 1283267776591424574 [12299.394740] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12299.409687] LustreError: 390544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [12310.221700] Lustre: server umount lustre-MDT0001 complete [12319.403298] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12323.405885] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [12325.938451] LustreError: 394961:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [12332.094686] Lustre: 395301:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.132@tcp [12334.119455] Lustre: server umount MGS complete [12345.779864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12346.335061] LustreError: 396027:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12346.365109] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12346.370879] Lustre: Skipped 1 previous similar message [12346.414459] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12350.694366] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [12351.458034] LustreError: 396028:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12358.791774] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12358.985320] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12363.352729] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [12371.634957] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12378.631526] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12380.978780] Lustre: Permanently reactivating lustre-OST0000 [12380.996445] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12381.005625] Lustre: Skipped 2 previous similar messages [12407.776430] LustreError: 397777:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12416.310704] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12416.702319] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12416.708297] Lustre: Skipped 1 previous similar message [12423.251964] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [12432.029400] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12438.504520] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12438.510723] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12438.511859] LustreError: Skipped 1 previous similar message [12438.512868] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12438.538991] Lustre: Skipped 2 previous similar messages [12439.723829] Lustre: server umount lustre-OST0000 complete [12446.176178] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12446.181793] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12450.411444] Lustre: server umount lustre-MDT0000 complete [12453.857775] LustreError: 396027:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12453.872422] LustreError: 396027:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12454.051283] LustreError: 396007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786640536 with bad export cookie 1283267776591425624 [12454.060226] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12454.061291] LustreError: 396007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12454.547373] Lustre: server umount lustre-MDT0001 complete [12467.224522] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [12470.318434] Key type lgssc unregistered [12470.764783] LNet: 400406:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12470.772393] LNetError: 400406:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12470.794303] LNet: Removed LNI 192.168.202.132@tcp [12471.746564] Key type .llcrypt unregistered [12471.749349] Key type ._llcrypt unregistered [12494.339448] Key type ._llcrypt registered [12494.346480] Key type .llcrypt registered [12494.573247] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [12511.804912] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [12513.195779] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12513.352855] alg: No test for adler32 (adler32-zlib) [12514.469478] Lustre: Lustre: Build Version: 2.17.57_2_g7a6d6aa [12514.812219] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [12516.519285] Key type lgssc registered [12517.887921] Lustre: Echo OBD driver; http://www.lustre.org/ [12557.369783] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12557.403117] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12558.828875] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12558.862839] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12558.916709] Lustre: lustre-MDT0000: new disk, initializing [12558.982181] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12559.001773] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12564.400348] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [12578.986382] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12579.150782] Lustre: 403575:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12579.243785] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12579.251512] Lustre: Skipped 1 previous similar message [12579.370435] Lustre: lustre-MDT0001: new disk, initializing [12579.506841] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12579.581218] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12579.602178] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12586.288929] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [12597.459372] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12605.486592] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12614.169518] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12614.565533] Lustre: lustre-OST0000: new disk, initializing [12614.574419] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12614.587121] Lustre: 405547:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12614.745167] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12621.959336] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [12622.993807] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12623.008953] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12623.161301] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12631.907786] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12636.684493] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12636.946847] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12642.507608] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12642.811495] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12648.961941] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12648.963516] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12648.970377] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12648.986015] LustreError: Skipped 1 previous similar message [12649.006408] Lustre: Skipped 1 previous similar message [12651.296531] Lustre: server umount lustre-OST0000 complete [12656.609099] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12656.620708] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12656.654928] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12656.674352] Lustre: Skipped 1 previous similar message [12659.179405] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12659.183063] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12661.906711] Lustre: server umount lustre-MDT0000 complete [12665.529311] LustreError: 403568:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786640747 with bad export cookie 5508279977559174415 [12665.543477] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12665.549181] LustreError: 403568:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12665.970565] Lustre: server umount lustre-MDT0001 complete [12676.080835] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12678.321186] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 13:05:57 (1786640757) [12685.722623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12686.114703] LustreError: 408008:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12686.135626] LustreError: 408008:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12686.193693] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12691.198438] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [12691.428175] LustreError: 408009:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12696.545431] LustreError: 408008:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12699.800244] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12700.232664] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12705.362775] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [12714.003525] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12720.497786] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12727.690668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12727.981677] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12734.582553] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [12742.824032] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12745.259320] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12748.344250] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12755.937730] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12755.949774] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12755.971327] Lustre: Skipped 1 previous similar message [12755.991274] Lustre: Skipped 2 previous similar messages [12761.059290] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12761.083218] Lustre: Skipped 1 previous similar message [12761.733940] Lustre: server umount lustre-OST0000 complete [12766.177768] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12766.209162] Lustre: Skipped 1 previous similar message [12771.298816] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12771.317496] Lustre: Skipped 3 previous similar messages [12772.377801] Lustre: server umount lustre-MDT0000 complete [12776.426200] LustreError: 408737:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12776.445707] LustreError: 408737:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12776.548980] LustreError: 407989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786640858 with bad export cookie 5508279977559175297 [12776.560111] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12776.917171] Lustre: server umount lustre-MDT0001 complete [12789.979600] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [12793.042468] Key type lgssc unregistered [12793.438627] LNet: 411737:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12793.446676] LNetError: 411737:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12793.466249] LNet: Removed LNI 192.168.202.132@tcp [12794.649150] Key type .llcrypt unregistered [12794.652073] Key type ._llcrypt unregistered [12821.123731] Key type ._llcrypt registered [12821.126324] Key type .llcrypt registered [12821.225821] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [12821.514494] Key type .llcrypt unregistered [12821.517775] Key type ._llcrypt unregistered [12843.134584] Key type ._llcrypt registered [12843.137626] Key type .llcrypt registered [12843.262878] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [12844.417582] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12844.587234] alg: No test for adler32 (adler32-zlib) [12845.627038] Lustre: Lustre: Build Version: 2.17.57_2_g7a6d6aa [12845.815218] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [12847.456514] Key type lgssc registered [12849.008974] Lustre: Echo OBD driver; http://www.lustre.org/ [12857.990154] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12858.035020] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12859.454551] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12864.789873] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [12869.603545] LustreError: 413956:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12869.621266] LustreError: 413956:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12873.999241] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12874.410732] LustreError: 413957:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12874.434025] LustreError: 413957:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12874.445825] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12879.850718] LustreError: 413956:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12879.865333] LustreError: 413956:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12880.906857] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [12884.963259] LustreError: 413952:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12884.984419] LustreError: 413952:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12884.994260] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12889.399806] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12890.084977] LustreError: 413952:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12890.102877] LustreError: 413952:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12896.858841] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12897.247118] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12902.097577] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [12912.612605] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12912.620965] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12912.628974] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12912.631144] Lustre: Skipped 1 previous similar message [12912.908602] Lustre: server umount lustre-OST0000 complete [12917.733291] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12917.739606] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12917.742227] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12917.742234] Lustre: Skipped 1 previous similar message [12917.755503] Lustre: Skipped 1 previous similar message [12917.793016] LustreError: Skipped 1 previous similar message [12921.285975] Lustre: server umount lustre-MDT0000 complete [12922.851524] LustreError: 413957:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12922.882407] LustreError: 413957:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12925.022100] LustreError: 413938:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786641007 with bad export cookie 12662429730861843257 [12925.030275] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12925.341822] Lustre: server umount lustre-MDT0001 complete [12937.457060] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [12940.878424] Key type lgssc unregistered [12941.172378] LNet: 417103:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12941.181849] LNetError: 417103:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12942.244818] LNet: Removed LNI 192.168.202.132@tcp [12943.134610] Key type .llcrypt unregistered [12943.137168] Key type ._llcrypt unregistered [12970.034100] Key type ._llcrypt registered [12970.037881] Key type .llcrypt registered [12970.293837] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [12972.152183] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12972.242780] alg: No test for adler32 (adler32-zlib) [12973.271644] Lustre: Lustre: Build Version: 2.17.57_2_g7a6d6aa [12973.615605] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [12975.263855] Key type lgssc registered [12976.668549] Lustre: Echo OBD driver; http://www.lustre.org/ [12985.492736] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12985.517727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13001.119494] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [13006.241291] LustreError: 418500:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.132@tcp: failed processing log, type 4: rc = -110 [13032.991179] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13039.077841] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [13045.971691] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13052.823189] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13053.343413] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13053.434448] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [13058.444941] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [13058.529555] LustreError: 418525:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13058.536926] LustreError: 418525:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13063.656704] LustreError: 418526:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13067.919981] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13068.424453] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13072.989880] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [13082.083573] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13087.965977] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13095.410790] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13095.429434] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13095.438151] Lustre: Skipped 1 previous similar message [13096.758563] Lustre: server umount lustre-OST0000 complete [13100.512256] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13100.521130] Lustre: Skipped 1 previous similar message [13100.526078] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13102.771287] Lustre: server umount lustre-MDT0000 complete [13105.633049] LustreError: 419544:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13105.659650] LustreError: 419544:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13105.867602] LustreError: 418507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786641187 with bad export cookie 8200751202974432673 [13105.876968] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13105.881292] LustreError: 418507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13106.167810] Lustre: server umount lustre-MDT0001 complete [13116.307683] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [13119.171855] Key type lgssc unregistered [13119.535948] LNet: 422073:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13119.542993] LNetError: 422073:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13119.555635] LNet: Removed LNI 192.168.202.132@tcp [13120.720927] Key type .llcrypt unregistered [13120.722445] Key type ._llcrypt unregistered [13144.848930] Key type ._llcrypt registered [13144.851926] Key type .llcrypt registered [13144.973845] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [13146.497341] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13146.519633] alg: No test for adler32 (adler32-zlib) [13147.561394] Lustre: Lustre: Build Version: 2.17.57_2_g7a6d6aa [13147.690873] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [13149.287197] Key type lgssc registered [13150.772047] Lustre: Echo OBD driver; http://www.lustre.org/ [13157.724783] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13157.745789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13159.201379] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13163.310918] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [13169.648514] LustreError: 423488:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13169.674395] LustreError: 423488:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13172.564292] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13172.882164] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13176.735675] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [13184.881581] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13190.865538] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13198.923732] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13199.254663] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13204.419735] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [13205.339891] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [13214.461398] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13221.925965] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.32@tcp (stopping) [13222.884519] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13222.888304] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13222.915230] Lustre: Skipped 1 previous similar message [13222.945112] Lustre: Skipped 2 previous similar messages [13226.974214] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.32@tcp (stopping) [13227.203638] Lustre: server umount lustre-MDT0000 complete [13228.009294] LustreError: 423488:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13228.031594] LustreError: 423488:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13231.122298] LustreError: 423470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.32@tcp arrived at 1786641313 with bad export cookie 15800797612580071566 [13233.126751] LustreError: 423485:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13233.166130] LustreError: 423485:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13233.959630] LustreError: 423469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786641315 with bad export cookie 15800797612580070677 [13233.980546] LustreError: 423469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13233.991909] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13238.242569] LustreError: 423488:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13238.259253] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13238.265173] LustreError: 423488:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13238.291610] Lustre: Skipped 1 previous similar message [13238.297229] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13240.388515] Lustre: server umount lustre-OST0000 complete [13243.362247] LustreError: 423489:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13243.376418] LustreError: 423489:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13249.567463] LustreError: 424217:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000f743eb77 x1873429113374592/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13249.918742] Lustre: server umount lustre-MDT0001 complete [13261.228260] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [13263.952736] Key type lgssc unregistered [13264.359542] LNet: 427177:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13264.373858] LNetError: 427177:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13264.394437] LNet: Removed LNI 192.168.202.132@tcp [13265.429950] Key type .llcrypt unregistered [13265.433440] Key type ._llcrypt unregistered [13292.141114] Key type ._llcrypt registered [13292.144838] Key type .llcrypt registered [13292.283764] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [13293.400598] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13293.526385] alg: No test for adler32 (adler32-zlib) [13294.793059] Lustre: Lustre: Build Version: 2.17.57_2_g7a6d6aa [13295.236411] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [13296.983816] Key type lgssc registered [13298.107144] Lustre: Echo OBD driver; http://www.lustre.org/ [13306.677927] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13306.699954] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13308.206364] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13312.933991] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [13318.636356] LustreError: 428594:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13318.665410] LustreError: 428594:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13321.230650] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13321.824920] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13327.128652] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [13340.240471] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13348.114572] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13357.762579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13358.157065] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13366.814054] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [13375.308345] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13379.660948] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [13387.792338] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.32@tcp (stopping) [13388.263352] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13388.269733] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13389.792313] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13389.794437] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13389.800742] Lustre: Skipped 1 previous similar message [13391.434560] Lustre: server umount lustre-MDT0001 complete [13392.892201] LustreError: 428589:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13394.919869] LustreError: 428594:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13394.929884] LustreError: 428594:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13400.034773] LustreError: 428595:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13400.046737] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13400.072211] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13400.080174] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13400.083115] Lustre: Skipped 1 previous similar message [13402.925474] Lustre: server umount lustre-OST0000 complete [13405.177434] LustreError: 428594:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13407.097587] Lustre: server umount lustre-MDT0000 complete [13421.840941] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [13424.197508] Key type lgssc unregistered [13424.468190] LNet: 432287:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13424.472688] LNetError: 432287:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13424.487670] LNet: Removed LNI 192.168.202.132@tcp [13425.444965] Key type .llcrypt unregistered [13425.448487] Key type ._llcrypt unregistered [13450.748403] Key type ._llcrypt registered [13450.751492] Key type .llcrypt registered [13450.843761] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [13466.636828] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [13468.254643] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13468.340677] alg: No test for adler32 (adler32-zlib) [13469.467464] Lustre: Lustre: Build Version: 2.17.57_2_g7a6d6aa [13469.803587] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [13471.495617] Key type lgssc registered [13472.701900] Lustre: Echo OBD driver; http://www.lustre.org/ [13519.594438] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [13536.784211] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13536.802644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13538.084772] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13538.113313] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13538.178463] Lustre: lustre-MDT0000: new disk, initializing [13538.232721] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13538.258346] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13545.416764] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [13564.766262] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13565.138149] Lustre: lustre-OST0000: new disk, initializing [13565.143923] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13565.148589] Lustre: Skipped 1 previous similar message [13565.163229] Lustre: 437346:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13565.267053] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13572.575797] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [13573.262527] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13573.276975] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13573.343338] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13581.489641] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13597.222681] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13597.313707] Lustre: 436328:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13597.402617] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13597.534691] Lustre: lustre-MDT0001: new disk, initializing [13597.697086] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13597.745025] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13597.759281] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13602.120359] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [13611.526769] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13611.719785] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13616.525263] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13616.749225] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13624.828816] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.32@tcp (stopping) [13626.850232] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13626.859562] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13626.860857] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13626.871526] Lustre: Skipped 1 previous similar message [13628.230424] Lustre: server umount lustre-MDT0001 complete [13637.102558] LustreError: 436339:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13637.131981] LustreError: 436339:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [13642.219395] LustreError: 436335:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13642.232180] LustreError: 436335:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13647.333080] LustreError: 436338:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13647.340403] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13647.384922] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13647.400601] Lustre: Skipped 1 previous similar message [13652.314832] Lustre: server umount lustre-OST0000 complete [13652.455947] LustreError: 436339:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13662.050543] Lustre: server umount lustre-MDT0000 complete [13675.913772] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [13678.438804] Key type lgssc unregistered [13678.713767] LNet: 441097:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13678.719858] LNetError: 441097:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13679.790968] LNet: Removed LNI 192.168.202.132@tcp [13680.572214] Key type .llcrypt unregistered [13680.575361] Key type ._llcrypt unregistered [13721.559968] Key type ._llcrypt registered [13721.561861] Key type .llcrypt registered [13721.649555] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [13721.956534] Key type .llcrypt unregistered [13721.959790] Key type ._llcrypt unregistered [13739.936708] Key type ._llcrypt registered [13739.938400] Key type .llcrypt registered [13740.051317] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [13741.930115] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13742.022603] alg: No test for adler32 (adler32-zlib) [13743.226571] Lustre: Lustre: Build Version: 2.17.57_2_g7a6d6aa [13743.566391] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [13745.247185] Key type lgssc registered [13746.476700] Lustre: Echo OBD driver; http://www.lustre.org/ [13780.480030] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13780.515382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13781.762359] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13781.790657] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13781.884950] Lustre: lustre-MDT0000: new disk, initializing [13781.964213] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13781.990793] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13787.843618] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [13799.270810] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13799.408294] Lustre: 444480:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13799.455400] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13799.462610] Lustre: Skipped 1 previous similar message [13799.572550] Lustre: lustre-MDT0001: new disk, initializing [13799.669882] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13799.744194] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13799.768596] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13805.351060] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [13816.202644] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13822.184820] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13829.236322] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13829.544476] Lustre: lustre-OST0000: new disk, initializing [13829.554566] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13829.562217] Lustre: 446450:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13829.668425] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13838.025310] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [13840.099526] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13840.120242] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13840.296959] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13851.891108] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13861.351213] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13861.379433] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13861.414281] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13865.956694] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13865.958476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13865.980088] Lustre: Skipped 1 previous similar message [13865.997936] Lustre: Skipped 2 previous similar messages [13867.515795] Lustre: server umount lustre-MDT0000 complete [13871.740091] LustreError: 444473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786641953 with bad export cookie 379398049833316599 [13871.751814] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13871.767592] LustreError: 444473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13872.292736] Lustre: server umount lustre-MDT0001 complete [13885.452124] Lustre: server umount lustre-OST0000 complete [13899.268160] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 13:26:18 (1786641978) [13911.282845] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13912.378622] LustreError: 448481:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13912.419537] LustreError: 448481:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [13912.564072] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13917.665697] LustreError: 448482:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13918.333825] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [13922.786052] LustreError: 448481:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13927.342989] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13927.689839] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13933.408328] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [13943.151164] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13950.345127] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13972.084964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13972.411195] Lustre: Found index 0 for lustre-OST0000, updating log [13972.700363] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13979.977788] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [13983.997782] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13988.979393] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13994.466207] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13994.467696] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13994.476165] LustreError: Skipped 1 previous similar message [13994.492183] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13994.503301] Lustre: Skipped 1 previous similar message [13999.906685] Lustre: server umount lustre-OST0000 complete [14016.392645] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14016.414093] Lustre: Skipped 1 previous similar message [14016.426734] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14016.429448] Lustre: Skipped 2 previous similar messages [14021.603483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14021.624526] Lustre: Skipped 1 previous similar message [14023.080663] Lustre: server umount lustre-MDT0000 complete [14027.754804] LustreError: 448481:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14027.784155] LustreError: 448481:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [14028.277566] LustreError: 448463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786642110 with bad export cookie 379398049833317992 [14028.283434] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14028.288758] LustreError: 448463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14028.865978] Lustre: server umount lustre-MDT0001 complete [14031.347191] LNet: 1 local NIs in recovery (showing 1): 192.168.202.132@tcp [14038.513772] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 13:28:38 (1786642118) [14659.176826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14659.651030] LustreError: 454893:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14659.714863] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14664.920343] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [14665.203661] LustreError: 454894:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14669.293335] LustreError: 454893:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14673.662124] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14674.090901] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14678.448755] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [14686.047240] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14692.347797] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14710.303643] LNet: 1 local NIs in recovery (showing 1): 192.168.202.132@tcp [14712.572333] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14712.797063] Lustre: Found index 0 for lustre-OST0000, updating log [14713.142494] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14719.209628] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [14727.047183] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14730.126187] Lustre: server umount lustre-OST0000 complete [14735.335220] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14735.343440] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14735.353427] Lustre: Skipped 1 previous similar message [14735.360114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14735.370447] Lustre: Skipped 2 previous similar messages [14736.434894] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14736.436435] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14738.541807] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14738.554887] Lustre: Skipped 4 previous similar messages [14741.199160] Lustre: server umount lustre-MDT0000 complete [14741.602741] LustreError: 456146:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14741.615482] LustreError: 456146:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [14745.175543] LustreError: 454874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786642827 with bad export cookie 379398049833319077 [14745.193025] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14745.706709] Lustre: server umount lustre-MDT0001 complete [14770.785692] LNet: 1 local NIs in recovery (showing 1): 192.168.202.132@tcp [14778.129898] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [14789.983898] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [14835.506603] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 13:41:54 (1786642914) [14853.739428] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [14856.894530] Key type lgssc unregistered [14857.305259] LNet: 461509:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14857.322491] LNetError: 461509:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14976.168193] LNet: Removed LNI 192.168.202.132@tcp [14977.072517] Key type .llcrypt unregistered [14977.074189] Key type ._llcrypt unregistered [14993.836264] Key type ._llcrypt registered [14993.838606] Key type .llcrypt registered [14993.989975] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [14995.199690] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14995.248591] alg: No test for adler32 (adler32-zlib) [14996.276934] Lustre: Lustre: Build Version: 2.17.57_2_g7a6d6aa [14996.498868] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [14998.159968] Key type lgssc registered [14999.532760] Lustre: Echo OBD driver; http://www.lustre.org/ [15004.137568] LNet: Added LNI 192.168.202.132@tcp100 [8/256/0/180] [15005.110121] LNet: Added LNI 192.168.202.132@tcp101 [8/256/0/180] [15006.062762] LNet: Added LNI 192.168.202.132@tcp102 [8/256/0/180] [15006.967991] LNet: Added LNI 192.168.202.132@tcp103 [8/256/0/180] [15007.739518] LNet: Added LNI 192.168.202.132@tcp104 [8/256/0/180] [15008.574974] LNet: Added LNI 192.168.202.132@tcp105 [8/256/0/180] [15009.627089] LNet: Added LNI 192.168.202.132@tcp106 [8/256/0/180] [15010.636677] LNet: Added LNI 192.168.202.132@tcp107 [8/256/0/180] [15011.628755] LNet: Added LNI 192.168.202.132@tcp108 [8/256/0/180] [15012.407433] LNet: Added LNI 192.168.202.132@tcp109 [8/256/0/180] [15013.327421] LNet: Added LNI 192.168.202.132@tcp110 [8/256/0/180] [15014.306153] LNet: Added LNI 192.168.202.132@tcp111 [8/256/0/180] [15015.226538] LNet: Added LNI 192.168.202.132@tcp112 [8/256/0/180] [15015.979933] LNet: Added LNI 192.168.202.132@tcp113 [8/256/0/180] [15017.098929] LNet: Added LNI 192.168.202.132@tcp114 [8/256/0/180] [15018.369988] LNet: Added LNI 192.168.202.132@tcp115 [8/256/0/180] [15019.669819] LNet: Added LNI 192.168.202.132@tcp116 [8/256/0/180] [15020.926266] LNet: Added LNI 192.168.202.132@tcp117 [8/256/0/180] [15022.155409] LNet: Added LNI 192.168.202.132@tcp118 [8/256/0/180] [15022.883902] LNet: Added LNI 192.168.202.132@tcp119 [8/256/0/180] [15023.676704] LNet: Added LNI 192.168.202.132@tcp120 [8/256/0/180] [15024.318519] LNet: Added LNI 192.168.202.132@tcp121 [8/256/0/180] [15025.419255] LNet: Added LNI 192.168.202.132@tcp122 [8/256/0/180] [15026.569505] LNet: Added LNI 192.168.202.132@tcp123 [8/256/0/180] [15027.589400] LNet: Added LNI 192.168.202.132@tcp124 [8/256/0/180] [15028.866794] LNet: Added LNI 192.168.202.132@tcp125 [8/256/0/180] [15030.273576] LNet: Added LNI 192.168.202.132@tcp126 [8/256/0/180] [15031.501655] LNet: Added LNI 192.168.202.132@tcp127 [8/256/0/180] [15032.397909] LNet: Added LNI 192.168.202.132@tcp128 [8/256/0/180] [15033.621048] LNet: Added LNI 192.168.202.132@tcp129 [8/256/0/180] [15034.795954] LNet: Added LNI 192.168.202.132@tcp130 [8/256/0/180] [15035.491062] LNet: Added LNI 192.168.202.132@tcp131 [8/256/0/180] [15036.150959] LNet: Added LNI 192.168.202.132@tcp132 [8/256/0/180] [15037.085364] LNet: Added LNI 192.168.202.132@tcp133 [8/256/0/180] [15037.952235] LNet: Added LNI 192.168.202.132@tcp134 [8/256/0/180] [15038.824769] LNet: Added LNI 192.168.202.132@tcp135 [8/256/0/180] [15045.561883] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15045.587541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15046.745398] Lustre: 464541:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.132@tcp: can't add NID 192.168.202.132@tcp130: rc = -75 [15046.833992] LNetError: 464567:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.132 [15046.850863] LNetError: Protocol error connecting to 192.168.202.132@tcp117 at host 192.168.202.132:988: is it running a compatible version of Lustre? [15046.945984] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15046.976188] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15047.054936] Lustre: lustre-MDT0000: new disk, initializing [15047.130250] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15047.152382] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15047.715965] LNet: 9 local NIs in recovery (showing 9): 192.168.202.132@tcp120, 192.168.202.132@tcp121, 192.168.202.132@tcp122, 192.168.202.132@tcp124, 192.168.202.132@tcp126, 192.168.202.132@tcp127, 192.168.202.132@tcp128, 192.168.202.132@tcp129, 192.168.202.132@tcp125 [15047.720424] LNetError: 464565:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.132 [15047.751787] LNetError: 464565:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 11 previous similar messages [15047.758678] LNetError: Protocol error connecting to 192.168.202.132@tcp119 at host 192.168.202.132:988: is it running a compatible version of Lustre? [15047.764335] LNetError: Skipped 14 previous similar messages [15049.827843] LNetError: 462001:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.132 [15049.849112] LNetError: 462001:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 7 previous similar messages [15049.861754] LNetError: Protocol error connecting to 192.168.202.132@tcp124 at host 192.168.202.132:988: is it running a compatible version of Lustre? [15049.873745] LNetError: Skipped 1 previous similar message [15051.823537] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [15053.923266] LNetError: 464570:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.132 [15053.942072] LNetError: Protocol error connecting to 192.168.202.132@tcp124 at host 192.168.202.132:988: is it running a compatible version of Lustre? [15061.093157] LNetError: 462000:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.132 [15061.098608] LNetError: Protocol error connecting to 192.168.202.132@tcp124 at host 192.168.202.132:988: is it running a compatible version of Lustre? [15061.873400] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15062.013197] Lustre: 464559:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15062.064257] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15062.066930] Lustre: Skipped 1 previous similar message [15062.129766] Lustre: lustre-MDT0001: new disk, initializing [15062.166324] Lustre: 465358:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.132@tcp: can't add NID 192.168.202.132@tcp130: rc = -75 [15062.176542] Lustre: 465358:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15062.276747] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15062.325265] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15062.342408] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15067.382392] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [15076.384296] LNetError: 461999:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.132 [15076.402266] LNetError: Protocol error connecting to 192.168.202.132@tcp123 at host 192.168.202.132:988: is it running a compatible version of Lustre? [15076.449760] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15082.453895] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15090.296740] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15090.612160] Lustre: lustre-OST0000: new disk, initializing [15090.622398] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15090.628625] Lustre: 466529:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15090.730848] Lustre: 466534:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.132@tcp: can't add NID 192.168.202.132@tcp130: rc = -75 [15090.736160] Lustre: 466534:0:(lustre_peer.c:124:class_add_uuid()) Skipped 17 previous similar messages [15090.773642] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15093.792286] LNetError: 462000:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.132 [15093.803788] LNetError: 462000:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 7 previous similar messages [15093.821747] LNetError: Protocol error connecting to 192.168.202.132@tcp116 at host 192.168.202.132:988: is it running a compatible version of Lustre? [15093.831238] LNetError: Skipped 7 previous similar messages [15096.796801] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [15098.892977] Lustre: 466987:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.132@tcp: can't add NID 192.168.202.132@tcp130: rc = -75 [15098.902913] Lustre: 466987:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15098.958117] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15098.967469] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15099.109310] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15105.789652] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15107.167167] LNet: 1 local NIs in recovery (showing 1): 192.168.202.132@tcp124 [15114.725468] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15114.728899] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15114.751217] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15114.760824] LustreError: Skipped 1 previous similar message [15114.779233] Lustre: Skipped 1 previous similar message [15118.809130] Lustre: server umount lustre-OST0000 complete [15123.943441] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15123.953155] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15123.963094] Lustre: Skipped 1 previous similar message [15123.982026] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15124.973375] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15124.996687] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15125.001935] Lustre: Skipped 1 previous similar message [15128.651604] Lustre: server umount lustre-MDT0000 complete [15134.086315] LNetError: 462001:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.132 [15134.091125] LNetError: 462001:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 13 previous similar messages [15134.095181] LNetError: Protocol error connecting to 192.168.202.132@tcp119 at host 192.168.202.132:988: is it running a compatible version of Lustre? [15134.100021] LNetError: Skipped 13 previous similar messages [15136.554941] LustreError: 464549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786643218 with bad export cookie 7273473867702284982 [15136.563694] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15136.580776] LustreError: 464549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15137.228760] Lustre: server umount lustre-MDT0001 complete [15153.630334] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [15158.324802] Key type lgssc unregistered [15158.720619] LNet: 468449:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15158.742461] LNetError: 468449:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15158.749265] LNet: Removed LNI 192.168.202.132@tcp [15159.775293] LNet: Removed LNI 192.168.202.132@tcp100 [15159.777448] LNetError: 468449:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15159.787356] LNetError: 468449:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15159.812262] LNet: Removed LNI 192.168.202.132@tcp101 [15160.863150] LNet: Removed LNI 192.168.202.132@tcp102 [15160.865629] LNetError: 468449:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15160.885512] LNetError: 468449:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15160.900710] LNet: Removed LNI 192.168.202.132@tcp103 [15161.951946] LNet: Removed LNI 192.168.202.132@tcp104 [15162.976677] LNet: Removed LNI 192.168.202.132@tcp105 [15162.990562] LNetError: 468449:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15163.022298] LNetError: 468449:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [15163.045462] LNet: Removed LNI 192.168.202.132@tcp106 [15164.067167] LNet: Removed LNI 192.168.202.132@tcp107 [15165.090025] LNet: Removed LNI 192.168.202.132@tcp108 [15166.111826] LNet: Removed LNI 192.168.202.132@tcp109 [15166.117227] LNet: Removed LNI 192.168.202.132@tcp110 [15167.135236] LNet: Removed LNI 192.168.202.132@tcp111 [15167.141449] LNetError: 468449:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15167.157557] LNetError: 468449:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [15167.176288] LNet: Removed LNI 192.168.202.132@tcp112 [15168.223289] LNet: Removed LNI 192.168.202.132@tcp113 [15169.247238] LNet: Removed LNI 192.168.202.132@tcp114 [15170.271559] LNet: Removed LNI 192.168.202.132@tcp115 [15171.295256] LNet: Removed LNI 192.168.202.132@tcp116 [15172.319227] LNet: Removed LNI 192.168.202.132@tcp117 [15173.343219] LNet: Removed LNI 192.168.202.132@tcp118 [15174.367685] LNet: Removed LNI 192.168.202.132@tcp119 [15175.391240] LNet: Removed LNI 192.168.202.132@tcp120 [15175.393674] LNetError: 468449:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15175.407304] LNetError: 468449:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [15175.420447] LNet: Removed LNI 192.168.202.132@tcp121 [15176.479219] LNet: Removed LNI 192.168.202.132@tcp122 [15177.503352] LNet: Removed LNI 192.168.202.132@tcp123 [15177.507514] LNet: Removed LNI 192.168.202.132@tcp124 [15178.528645] LNet: Removed LNI 192.168.202.132@tcp125 [15179.552156] LNet: Removed LNI 192.168.202.132@tcp126 [15180.575625] LNet: Removed LNI 192.168.202.132@tcp127 [15181.599320] LNet: Removed LNI 192.168.202.132@tcp128 [15182.623146] LNet: Removed LNI 192.168.202.132@tcp129 [15182.625288] LNet: Removed LNI 192.168.202.132@tcp130 [15182.627786] LNet: Removed LNI 192.168.202.132@tcp131 [15182.633852] LNet: Removed LNI 192.168.202.132@tcp132 [15182.637886] LNet: Removed LNI 192.168.202.132@tcp133 [15182.651131] LNet: Removed LNI 192.168.202.132@tcp134 [15182.664769] LNet: Removed LNI 192.168.202.132@tcp135 [15183.605532] Key type .llcrypt unregistered [15183.608351] Key type ._llcrypt unregistered [15209.866480] Key type ._llcrypt registered [15209.872327] Key type .llcrypt registered [15210.023871] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [15225.541857] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [15226.821597] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15226.833134] alg: No test for adler32 (adler32-zlib) [15227.857378] Lustre: Lustre: Build Version: 2.17.57_2_g7a6d6aa [15228.077965] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [15229.736109] Key type lgssc registered [15231.259382] Lustre: Echo OBD driver; http://www.lustre.org/ [15270.668099] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 13:49:10 (1786643350) [15292.907906] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [15295.064887] Key type lgssc unregistered [15295.305234] LNet: 472541:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15295.311292] LNetError: 472541:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15295.327469] LNet: Removed LNI 192.168.202.132@tcp [15296.260789] Key type .llcrypt unregistered [15296.270556] Key type ._llcrypt unregistered [15312.238877] Key type ._llcrypt registered [15312.240661] Key type .llcrypt registered [15312.314408] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [15313.292236] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15313.373436] alg: No test for adler32 (adler32-zlib) [15314.452845] Lustre: Lustre: Build Version: 2.17.57_2_g7a6d6aa [15314.712435] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [15316.415175] Key type lgssc registered [15318.093960] Lustre: Echo OBD driver; http://www.lustre.org/ [15327.030300] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15327.045858] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15328.307314] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15328.327866] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15328.394681] Lustre: lustre-MDT0000: new disk, initializing [15328.457796] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15328.472790] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15332.583722] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [15342.623040] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15342.749798] Lustre: 473754:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15342.817785] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15342.822592] Lustre: Skipped 1 previous similar message [15342.904877] Lustre: lustre-MDT0001: new disk, initializing [15343.005401] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15343.038521] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15343.060296] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15347.411481] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [15356.724164] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15363.884803] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15371.784494] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15372.072495] Lustre: lustre-OST0000: new disk, initializing [15372.075769] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15372.083312] Lustre: 475723:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15372.186597] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15378.245751] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [15381.060317] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15381.076366] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15381.147518] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15387.870484] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15391.672218] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing lnet_if_list [15397.079870] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [15397.095882] LNet: Added LNI 192.168.202.132@tcp10 [8/256/0/180] [15402.404183] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [15402.440600] LNet: Added LNI 192.168.202.132@tcp11 [8/256/0/180] [15406.776637] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [15406.814649] LNet: Added LNI 192.168.202.132@tcp12 [8/256/0/180] [15411.748224] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [15411.771306] LNet: Added LNI 192.168.202.132@tcp13 [8/256/0/180] [15416.840740] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [15416.875866] LNet: Added LNI 192.168.202.132@tcp14 [8/256/0/180] [15421.881785] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [15421.914977] LNet: Added LNI 192.168.202.132@tcp15 [8/256/0/180] [15428.197917] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [15428.280976] LNet: Added LNI 192.168.202.132@tcp16 [8/256/0/180] [15435.343538] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [15435.391859] LNet: Added LNI 192.168.202.132@tcp17 [8/256/0/180] [15443.514529] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [15443.547325] LNet: Added LNI 192.168.202.132@tcp18 [8/256/0/180] [15450.684177] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [15450.695792] LNet: Added LNI 192.168.202.132@tcp19 [8/256/0/180] [15457.493252] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [15457.527648] LNet: Added LNI 192.168.202.132@tcp20 [8/256/0/180] [15463.907814] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [15463.975547] LNet: Added LNI 192.168.202.132@tcp21 [8/256/0/180] [15469.891874] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [15469.911535] LNet: Added LNI 192.168.202.132@tcp22 [8/256/0/180] [15476.254915] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [15476.284468] LNet: Added LNI 192.168.202.132@tcp23 [8/256/0/180] [15482.095701] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [15482.176306] LNet: Added LNI 192.168.202.132@tcp24 [8/256/0/180] [15489.234976] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [15489.262755] LNet: Added LNI 192.168.202.132@tcp25 [8/256/0/180] [15495.916854] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [15495.940910] LNet: Added LNI 192.168.202.132@tcp26 [8/256/0/180] [15502.594897] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [15502.658646] LNet: Added LNI 192.168.202.132@tcp27 [8/256/0/180] [15514.480696] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [15514.551972] LNet: Added LNI 192.168.202.132@tcp28 [8/256/0/180] [15523.696778] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [15523.719848] LNet: Added LNI 192.168.202.132@tcp29 [8/256/0/180] [15531.483497] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [15531.510484] LNet: Added LNI 192.168.202.132@tcp30 [8/256/0/180] [15538.497433] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [15538.527756] LNet: Added LNI 192.168.202.132@tcp31 [8/256/0/180] [15545.076270] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [15545.095646] LNet: Added LNI 192.168.202.132@tcp32 [8/256/0/180] [15550.843383] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [15550.868474] LNet: Added LNI 192.168.202.132@tcp33 [8/256/0/180] [15556.600401] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [15556.620966] LNet: Added LNI 192.168.202.132@tcp34 [8/256/0/180] [15562.900281] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [15562.924699] LNet: Added LNI 192.168.202.132@tcp35 [8/256/0/180] [15569.072393] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [15569.097787] LNet: Added LNI 192.168.202.132@tcp36 [8/256/0/180] [15577.000895] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [15577.017557] LNet: Added LNI 192.168.202.132@tcp37 [8/256/0/180] [15584.997315] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [15585.028274] LNet: Added LNI 192.168.202.132@tcp38 [8/256/0/180] [15592.701404] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [15592.729357] LNet: Added LNI 192.168.202.132@tcp39 [8/256/0/180] [15599.509614] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [15599.550845] LNet: Added LNI 192.168.202.132@tcp40 [8/256/0/180] [15606.885496] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [15606.908598] LNet: Added LNI 192.168.202.132@tcp41 [8/256/0/180] [15613.960113] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [15613.993826] LNet: Added LNI 192.168.202.132@tcp42 [8/256/0/180] [15621.572078] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [15621.598038] LNet: Added LNI 192.168.202.132@tcp43 [8/256/0/180] [15628.493530] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [15628.513774] LNet: Added LNI 192.168.202.132@tcp44 [8/256/0/180] [15635.691670] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [15635.754130] LNet: Added LNI 192.168.202.132@tcp45 [8/256/0/180] [15644.743931] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [15644.766823] LNet: Added LNI 192.168.202.132@tcp46 [8/256/0/180] [15652.407865] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [15652.434903] LNet: Added LNI 192.168.202.132@tcp47 [8/256/0/180] [15661.402197] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [15661.441190] LNet: Added LNI 192.168.202.132@tcp48 [8/256/0/180] [15670.936405] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [15670.965405] LNet: Added LNI 192.168.202.132@tcp49 [8/256/0/180] [15683.031803] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [15683.064552] LNet: Added LNI 192.168.202.132@tcp50 [8/256/0/180] [15706.592169] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15706.604168] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15706.620604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15709.154989] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15709.157139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15709.168210] Lustre: Skipped 1 previous similar message [15709.176257] Lustre: Skipped 2 previous similar messages [15714.272704] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15714.282053] Lustre: Skipped 2 previous similar messages [15719.391198] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15719.398346] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15719.410758] Lustre: Skipped 2 previous similar messages [15719.544380] Lustre: server umount lustre-MDT0000 complete [15723.093082] LustreError: 473745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786643805 with bad export cookie 3222602747306398794 [15723.097334] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15723.105265] LustreError: 473745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15723.772306] Lustre: server umount lustre-MDT0001 complete [15740.893480] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [15740.897273] Lustre: 473026:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786643806/real 1786643806] req@0000000043aef6c7 x1873431385782656/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786643822 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15740.934090] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15744.992695] Lustre: 473028:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786643811/real 1786643811] req@00000000657c1531 x1873431385782912/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786643827 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15745.016141] Lustre: 473028:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [15748.915251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15749.088244] Lustre: 473029:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786643815/real 1786643815] req@00000000f032b349 x1873431385783168/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786643831 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15754.143241] Lustre: 473027:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786643820/real 1786643820] req@00000000cf3d4502 x1873431385783296/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786643836 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15768.543763] LustreError: 473025:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000cf3d4502 x1873431385784064/t0(0) o250->MGC192.168.202.132@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15768.781421] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15772.562969] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [15774.883573] LustreError: 475718:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15774.888516] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15774.894228] LustreError: 475718:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [15780.332706] LustreError: 475717:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15780.468133] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15780.846183] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15785.375663] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [15785.978676] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15791.792882] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15796.555690] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15815.142133] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15815.159709] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15815.179938] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15816.685462] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15820.734879] Lustre: server umount lustre-OST0000 complete [15826.912045] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15826.914553] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15826.920742] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15826.923851] Lustre: Skipped 2 previous similar messages [15826.928173] Lustre: Skipped 2 previous similar messages [15830.680817] Lustre: server umount lustre-MDT0000 complete [15832.048475] LustreError: 499966:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15832.082820] LustreError: 499966:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [15834.250850] LustreError: 475733:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786643916 with bad export cookie 3222602747306401391 [15834.271895] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15834.707933] Lustre: server umount lustre-MDT0001 complete [15845.742698] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [15848.120556] Key type lgssc unregistered [15848.495497] LNet: 502748:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15848.506531] LNetError: 502748:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15849.568268] LNet: Removed LNI 192.168.202.132@tcp [15849.570637] LNetError: 502748:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15849.576858] LNet: Removed LNI 192.168.202.132@tcp10 [15849.581066] LNet: Removed LNI 192.168.202.132@tcp11 [15849.586355] LNet: Removed LNI 192.168.202.132@tcp12 [15849.588463] LNet: Removed LNI 192.168.202.132@tcp13 [15849.597457] LNet: Removed LNI 192.168.202.132@tcp14 [15849.602394] LNet: Removed LNI 192.168.202.132@tcp15 [15849.606630] LNet: Removed LNI 192.168.202.132@tcp16 [15849.609141] LNet: Removed LNI 192.168.202.132@tcp17 [15849.611347] LNet: Removed LNI 192.168.202.132@tcp18 [15849.614191] LNet: Removed LNI 192.168.202.132@tcp19 [15849.616343] LNet: Removed LNI 192.168.202.132@tcp20 [15849.618806] LNet: Removed LNI 192.168.202.132@tcp21 [15849.621126] LNet: Removed LNI 192.168.202.132@tcp22 [15849.623378] LNet: Removed LNI 192.168.202.132@tcp23 [15849.625017] LNet: Removed LNI 192.168.202.132@tcp24 [15849.630638] LNet: Removed LNI 192.168.202.132@tcp25 [15849.635311] LNet: Removed LNI 192.168.202.132@tcp26 [15849.638368] LNet: Removed LNI 192.168.202.132@tcp27 [15849.641430] LNet: Removed LNI 192.168.202.132@tcp28 [15849.643980] LNet: Removed LNI 192.168.202.132@tcp29 [15849.648029] LNet: Removed LNI 192.168.202.132@tcp30 [15849.650928] LNet: Removed LNI 192.168.202.132@tcp31 [15849.654084] LNet: Removed LNI 192.168.202.132@tcp32 [15849.657166] LNet: Removed LNI 192.168.202.132@tcp33 [15849.660099] LNet: Removed LNI 192.168.202.132@tcp34 [15849.663467] LNet: Removed LNI 192.168.202.132@tcp35 [15849.666784] LNet: Removed LNI 192.168.202.132@tcp36 [15849.669228] LNet: Removed LNI 192.168.202.132@tcp37 [15849.672343] LNet: Removed LNI 192.168.202.132@tcp38 [15849.675453] LNet: Removed LNI 192.168.202.132@tcp39 [15849.681976] LNet: Removed LNI 192.168.202.132@tcp40 [15849.688084] LNet: Removed LNI 192.168.202.132@tcp41 [15849.690175] LNet: Removed LNI 192.168.202.132@tcp42 [15849.692848] LNet: Removed LNI 192.168.202.132@tcp43 [15849.698939] LNet: Removed LNI 192.168.202.132@tcp44 [15849.701128] LNet: Removed LNI 192.168.202.132@tcp45 [15849.703032] LNet: Removed LNI 192.168.202.132@tcp46 [15849.705594] LNet: Removed LNI 192.168.202.132@tcp47 [15849.707589] LNet: Removed LNI 192.168.202.132@tcp48 [15849.709889] LNet: Removed LNI 192.168.202.132@tcp49 [15849.720848] LNet: Removed LNI 192.168.202.132@tcp50 [15850.443404] Key type .llcrypt unregistered [15850.444983] Key type ._llcrypt unregistered [15866.903823] Key type ._llcrypt registered [15866.906475] Key type .llcrypt registered [15866.994602] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [15881.091662] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [15882.159548] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15882.417243] alg: No test for adler32 (adler32-zlib) [15883.650274] Lustre: Lustre: Build Version: 2.17.57_2_g7a6d6aa [15883.973731] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [15885.791233] Key type lgssc registered [15886.758618] Lustre: Echo OBD driver; http://www.lustre.org/ [15921.895784] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15921.913992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15923.280074] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15923.320533] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15923.377139] Lustre: lustre-MDT0000: new disk, initializing [15923.485575] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15923.506522] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15927.211708] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [15937.108873] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15937.306818] Lustre: 505944:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15937.384332] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15937.388709] Lustre: Skipped 1 previous similar message [15937.511184] Lustre: lustre-MDT0001: new disk, initializing [15937.623797] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15937.657535] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15937.673630] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15942.234643] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [15951.074981] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15955.559773] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15962.447219] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15962.616817] Lustre: *** cfs_fail_loc=90f, val=0*** [15962.674266] Lustre: lustre-OST0000: new disk, initializing [15962.676804] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15962.682426] Lustre: 507961:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15962.752821] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15969.009382] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [15972.944204] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15972.956384] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15973.108174] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15978.126034] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15988.707417] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15988.710185] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15988.718522] LustreError: Skipped 1 previous similar message [15988.720539] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15988.731039] Lustre: Skipped 1 previous similar message [15988.744950] Lustre: Skipped 1 previous similar message [15991.460574] Lustre: server umount lustre-OST0000 complete [15998.948570] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15998.948899] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15998.959980] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15998.970527] Lustre: Skipped 1 previous similar message [16001.258337] Lustre: server umount lustre-MDT0000 complete [16004.497474] LustreError: 505936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786644086 with bad export cookie 5954224609508355827 [16004.510960] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16004.514931] LustreError: 505936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [16004.806095] Lustre: server umount lustre-MDT0001 complete [16016.627423] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [16020.033488] Key type lgssc unregistered [16020.390627] LNet: 509880:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16020.399543] LNetError: 509880:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16020.428142] LNet: Removed LNI 192.168.202.132@tcp [16021.814601] Key type .llcrypt unregistered [16021.816616] Key type ._llcrypt unregistered [16040.155269] Key type ._llcrypt registered [16040.158263] Key type .llcrypt registered [16040.268973] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [16041.299263] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16041.339880] alg: No test for adler32 (adler32-zlib) [16042.347575] Lustre: Lustre: Build Version: 2.17.57_2_g7a6d6aa [16042.687216] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [16044.407233] Key type lgssc registered [16045.442512] Lustre: Echo OBD driver; http://www.lustre.org/ [16052.581977] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16052.599802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16054.105488] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16057.561120] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [16064.481665] LustreError: 511144:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16064.504105] LustreError: 511144:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [16065.201866] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16065.487808] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16069.712514] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [16076.634104] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16081.786788] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16087.940035] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16088.234427] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16094.069519] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [16102.115386] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16112.254189] loop0: detected capacity change from 0 to 204800000 [16112.464808] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16112.472972] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16112.481396] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16118.453376] loop0: detected capacity change from 0 to 204800000 [16118.614496] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16118.624153] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16118.635238] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16123.891391] loop0: detected capacity change from 0 to 204800000 [16129.051418] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16129.242829] Lustre: fs15246-OST0000: new disk, initializing [16129.247683] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [16129.252911] Lustre: 514736:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16129.308961] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16135.080236] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [16143.470764] loop1: detected capacity change from 0 to 204800000 [16148.759301] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16148.917616] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [16148.975879] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [16149.098537] Lustre: fs15246-MDT0000: new disk, initializing [16149.243287] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16149.269604] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16149.387510] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [16150.437388] Lustre: 515856:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@000000007bfa5afc x1873432149012224/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1786644248 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [16154.656599] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [16159.522565] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [16159.533087] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [16159.555180] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [16178.154522] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16178.188466] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [16182.186837] Lustre: server umount fs15246-MDT0000 complete [16183.296796] LustreError: 511140:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16188.395292] LustreError: 511140:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16191.738253] Lustre: server umount fs15246-OST0000 complete [16203.746849] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16203.781102] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16208.363767] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16208.382866] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16208.414805] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16209.976460] Lustre: server umount lustre-OST0000 complete [16219.111775] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16219.112387] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16219.112628] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16219.112633] Lustre: Skipped 1 previous similar message [16219.132926] Lustre: Skipped 1 previous similar message [16220.784508] Lustre: server umount lustre-MDT0000 complete [16224.233713] LustreError: 511144:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16224.843563] LustreError: 511879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786644306 with bad export cookie 3661696010578232946 [16224.846115] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16224.870968] LustreError: 511879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16225.306918] Lustre: server umount lustre-MDT0001 complete [16238.239213] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [16241.847207] Key type lgssc unregistered [16242.131738] LNet: 518955:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16242.143938] LNetError: 518955:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16242.188019] LNet: Removed LNI 192.168.202.132@tcp [16243.313200] Key type .llcrypt unregistered [16243.315877] Key type ._llcrypt unregistered [16259.474674] Key type ._llcrypt registered [16259.476204] Key type .llcrypt registered [16259.575290] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [16260.381398] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16260.501090] alg: No test for adler32 (adler32-zlib) [16261.667122] Lustre: Lustre: Build Version: 2.17.57_2_g7a6d6aa [16261.885359] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [16263.543472] Key type lgssc registered [16265.019352] Lustre: Echo OBD driver; http://www.lustre.org/ [16300.063455] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [16309.943151] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [16348.400131] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16348.438706] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16349.851387] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16349.903784] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16350.050560] Lustre: lustre-MDT0000: new disk, initializing [16350.193027] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16350.216489] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16354.934905] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [16365.824969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16366.020545] Lustre: 523414:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16366.169899] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16366.174910] Lustre: Skipped 1 previous similar message [16366.370582] Lustre: lustre-MDT0001: new disk, initializing [16366.549707] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16366.602958] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16366.615431] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16371.036492] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [16382.201815] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16389.860095] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16398.400706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16398.960639] Lustre: lustre-OST0000: new disk, initializing [16398.964500] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16398.972830] Lustre: 525388:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16399.103032] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16407.201793] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16407.214696] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16407.357951] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16407.865839] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [16419.198527] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16425.525445] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [16425.702272] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [16431.170105] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [16431.366846] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [16436.193853] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16436.196304] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16436.202028] Lustre: Skipped 1 previous similar message [16440.210810] Lustre: server umount lustre-OST0000 complete [16446.434508] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16446.469623] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16446.472114] Lustre: Skipped 1 previous similar message [16451.864629] Lustre: server umount lustre-MDT0000 complete [16456.677351] LustreError: 524535:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16456.699943] LustreError: 524535:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [16457.325154] LustreError: 523407:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786644539 with bad export cookie 15911498097055565832 [16457.331924] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16457.337262] LustreError: 523407:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16457.962401] Lustre: server umount lustre-MDT0001 complete [16487.298864] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [16497.436364] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [16539.518942] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 16257 sec ========= 14:10:18 (1786644618) [16541.726825] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 14:10:20 (1786644620) === [16545.390882] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 14:10:25 (1786644625) === [16590.317545] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [16593.804661] Key type lgssc unregistered [16594.211989] LNet: 532037:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16594.221435] LNetError: 532037:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16594.244231] LNet: Removed LNI 192.168.202.132@tcp [16595.121544] Key type .llcrypt unregistered [16595.124434] Key type ._llcrypt unregistered