[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 563982221 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002395] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004014] kvm-guest: setup PV IPIs [ 0.007350] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.010133] LSM: Security Framework initializing [ 0.011060] Yama: becoming mindful. [ 0.012041] SELinux: Initializing. [ 0.013072] *** VALIDATE selinux *** [ 0.022782] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027422] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028161] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029117] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030160] *** VALIDATE tmpfs *** [ 0.032285] *** VALIDATE proc *** [ 0.033259] *** VALIDATE cgroup *** [ 0.034010] *** VALIDATE cgroup2 *** [ 0.035279] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036138] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037014] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038030] Spectre V2 : User space: Vulnerable [ 0.039020] Speculative Store Bypass: Vulnerable [ 0.042169] debug: unmapping init [mem 0xffffffff91459000-0xffffffff91460fff] [ 0.045240] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046997] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048028] ... version: 2 [ 0.049015] ... bit width: 48 [ 0.050013] ... generic registers: 4 [ 0.051014] ... value mask: 0000ffffffffffff [ 0.052015] ... max period: 00007fffffffffff [ 0.053177] ... fixed-purpose events: 3 [ 0.054131] ... event mask: 000000070000000f [ 0.055298] rcu: Hierarchical SRCU implementation. [ 0.057774] smp: Bringing up secondary CPUs ... [ 0.058728] x86: Booting SMP configuration: [ 0.059053] .... node #0, CPUs: #1 #2 #3 [ 0.070166] smp: Brought up 1 node, 4 CPUs [ 0.072031] smpboot: Max logical packages: 1 [ 0.073015] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.108640] node 0 deferred pages initialised in 33ms [ 0.112239] devtmpfs: initialized [ 0.113252] x86/mm: Memory block size: 128MB [ 0.116851] gcov: version magic: 0x41383552 [ 0.118290] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.119234] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.120250] pinctrl core: initialized pinctrl subsystem [ 0.121337] [ 0.121839] ************************************************************* [ 0.122015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.123020] ** ** [ 0.124015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.125016] ** ** [ 0.126015] ** This means that this kernel is built to expose internal ** [ 0.127024] ** IOMMU data structures, which may compromise security on ** [ 0.128019] ** your system. ** [ 0.129014] ** ** [ 0.130172] ** If you see this message and you are not debugging the ** [ 0.131016] ** kernel, report this immediately to your vendor! ** [ 0.132015] ** ** [ 0.133013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.134012] ************************************************************* [ 0.135863] NET: Registered protocol family 16 [ 0.136527] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.137077] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.138076] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.140060] cpuidle: using governor menu [ 0.143116] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.147869] PCI: Using configuration type 1 for base access [ 0.152193] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.168380] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.171020] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.175127] cryptd: max_cpu_qlen set to 1000 [ 0.177431] ACPI: Added _OSI(Module Device) [ 0.179011] ACPI: Added _OSI(Processor Device) [ 0.181013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.183051] ACPI: Added _OSI(Processor Aggregator Device) [ 0.190749] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.198883] ACPI: Interpreter enabled [ 0.201059] ACPI: PM: (supports S0 S3 S4 S5) [ 0.203013] ACPI: Using IOAPIC for interrupt routing [ 0.206118] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.211419] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.225463] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.229040] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.232017] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.237074] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.244329] acpiphp: Slot [2] registered [ 0.248217] acpiphp: Slot [5] registered [ 0.251393] acpiphp: Slot [6] registered [ 0.255413] acpiphp: Slot [7] registered [ 0.258807] acpiphp: Slot [8] registered [ 0.262327] acpiphp: Slot [9] registered [ 0.264135] acpiphp: Slot [10] registered [ 0.266105] acpiphp: Slot [3] registered [ 0.268138] acpiphp: Slot [4] registered [ 0.270436] acpiphp: Slot [11] registered [ 0.272229] acpiphp: Slot [12] registered [ 0.275120] acpiphp: Slot [13] registered [ 0.277151] acpiphp: Slot [14] registered [ 0.280118] acpiphp: Slot [15] registered [ 0.282125] acpiphp: Slot [16] registered [ 0.284119] acpiphp: Slot [17] registered [ 0.286109] acpiphp: Slot [18] registered [ 0.288136] acpiphp: Slot [19] registered [ 0.290268] acpiphp: Slot [20] registered [ 0.292224] acpiphp: Slot [21] registered [ 0.295153] acpiphp: Slot [22] registered [ 0.297117] acpiphp: Slot [23] registered [ 0.300230] acpiphp: Slot [24] registered [ 0.302199] acpiphp: Slot [25] registered [ 0.304198] acpiphp: Slot [26] registered [ 0.307127] acpiphp: Slot [27] registered [ 0.309104] acpiphp: Slot [28] registered [ 0.311084] acpiphp: Slot [29] registered [ 0.313132] acpiphp: Slot [30] registered [ 0.315183] acpiphp: Slot [31] registered [ 0.317086] PCI host bridge to bus 0000:00 [ 0.319018] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.322019] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.326119] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.330021] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.334041] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.338020] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.339337] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.342200] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.345385] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.356934] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.362047] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.364060] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.366015] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.370018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.372401] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.374679] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.378041] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.380844] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.386014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.390000] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.390000] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.390000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.390000] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.390000] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.402019] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.476530] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.519018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.530016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.562015] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.575653] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.584016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.593015] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.625015] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.638292] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.651013] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.660015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.683024] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.696987] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.706013] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.717015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.737015] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.738000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.738000] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.747031] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.769013] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.780895] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.782479] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.785350] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.788334] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.791260] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.797098] iommu: Default domain type: Passthrough [ 0.799399] SCSI subsystem initialized [ 0.801170] ACPI: bus type USB registered [ 0.802182] usbcore: registered new interface driver usbfs [ 0.804074] usbcore: registered new interface driver hub [ 0.806075] usbcore: registered new device driver usb [ 0.808199] pps_core: LinuxPPS API ver. 1 registered [ 0.810011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.813052] PTP clock support registered [ 0.815331] EDAC MC: Ver: 3.0.0 [ 0.817079] PCI: Using ACPI for IRQ routing [ 0.820023] NetLabel: Initializing [ 0.821008] NetLabel: domain hash size = 128 [ 0.823010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.825127] NetLabel: unlabeled traffic allowed by default [ 0.828401] vgaarb: loaded [ 0.831423] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.833009] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.837240] clocksource: Switched to clocksource kvm-clock [ 0.980385] VFS: Disk quotas dquot_6.6.0 [ 0.981780] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.984667] *** VALIDATE ramfs *** [ 0.985975] *** VALIDATE hugetlbfs *** [ 0.987652] pnp: PnP ACPI init [ 0.990256] pnp: PnP ACPI: found 6 devices [ 1.007314] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.010717] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.013194] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.015130] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.018194] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.020955] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.024365] NET: Registered protocol family 2 [ 1.027312] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.033693] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.037574] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.042447] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.045758] TCP: Hash tables configured (established 65536 bind 65536) [ 1.048932] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.052778] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.056225] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.059637] NET: Registered protocol family 1 [ 1.063172] RPC: Registered named UNIX socket transport module. [ 1.065564] RPC: Registered udp transport module. [ 1.067450] RPC: Registered tcp transport module. [ 1.069210] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.071846] NET: Registered protocol family 44 [ 1.073860] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.076418] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.078239] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.080561] PCI: CLS 0 bytes, default 64 [ 1.082448] Unpacking initramfs... [ 2.921382] debug: unmapping init [mem 0xffff8c5f7cc54000-0xffff8c5f7ffbffff] [ 2.927203] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.929862] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.933880] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.937069] Initialise system trusted keyrings [ 3.946423] Key type blacklist registered [ 3.975346] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.988687] zbud: loaded [ 3.993930] *** VALIDATE nfs *** [ 3.996546] *** VALIDATE nfs4 *** [ 3.999237] pstore: using deflate compression [ 4.003689] Platform Keyring initialized [ 4.259611] NET: Registered protocol family 38 [ 4.261047] Key type asymmetric registered [ 4.262705] Asymmetric key parser 'x509' registered [ 4.265626] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.269423] io scheduler mq-deadline registered [ 4.271435] io scheduler kyber registered [ 4.273562] io scheduler bfq registered [ 4.275450] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.279075] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.282959] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.286705] ACPI: Power Button [PWRF] [ 4.294976] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.303375] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.318193] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.328644] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.345578] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.374365] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.402444] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.414588] Non-volatile memory driver v1.3 [ 4.416684] Linux agpgart interface v0.103 [ 4.458543] virtio_blk virtio1: [vda] 134136 512-byte logical blocks (68.7 MB/65.5 MiB) [ 4.462563] vda: detected capacity change from 0 to 68677632 [ 4.478353] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.481558] vdb: detected capacity change from 0 to 1073741824 [ 4.546369] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.549572] vdc: detected capacity change from 0 to 2621440000 [ 4.599421] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.602924] vdd: detected capacity change from 0 to 2621440000 [ 4.637288] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.640297] vde: detected capacity change from 0 to 4294967296 [ 4.655693] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.658760] vdf: detected capacity change from 0 to 4294967296 [ 4.665717] libphy: Fixed MDIO Bus: probed [ 4.693343] usbcore: registered new interface driver usbserial_generic [ 4.696910] usbserial: USB Serial support registered for generic [ 4.700670] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.708469] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.710723] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.713819] mousedev: PS/2 mouse device common for all mice [ 4.717842] rtc_cmos 00:05: RTC can wake from S4 [ 4.721578] rtc_cmos 00:05: registered as rtc0 [ 4.724429] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.728371] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.732141] intel_pstate: CPU model not supported [ 4.734675] hid: raw HID events driver (C) Jiri Kosina [ 4.745984] usbcore: registered new interface driver usbhid [ 4.748608] usbhid: USB HID core driver [ 4.750488] drop_monitor: Initializing network drop monitor service [ 4.755283] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.756304] Initializing XFRM netlink socket [ 4.763983] NET: Registered protocol family 10 [ 4.767563] Segment Routing with IPv6 [ 4.769690] NET: Registered protocol family 17 [ 4.772023] mpls_gso: MPLS GSO support [ 4.773271] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.784640] RAS: Correctable Errors collector initialized. [ 4.787475] AVX version of gcm_enc/dec engaged. [ 4.789429] AES CTR mode by8 optimization enabled [ 4.893976] sched_clock: Marking stable (4893954533, 0)->(6293934117, -1399979584) [ 4.898866] registered taskstats version 1 [ 4.901777] Loading compiled-in X.509 certificates [ 4.904273] zswap: loaded using pool lzo/zbud [ 4.933178] Key type big_key registered [ 4.948230] Key type encrypted registered [ 4.951935] ima: No TPM chip found, activating TPM-bypass! [ 4.954964] ima: Allocated hash algorithm: sha1 [ 4.957339] ima: No architecture policies found [ 4.959815] evm: Initialising EVM extended attributes: [ 4.962301] evm: security.selinux [ 4.963793] evm: security.ima [ 4.965317] evm: security.capability [ 4.967201] evm: HMAC attrs: 0x1 [ 4.969898] rtc_cmos 00:05: setting system clock to 2026-04-10 18:42:47 UTC (1775846567) [ 4.977477] debug: unmapping init [mem 0xffffffff92403000-0xffffffff925fffff] [ 4.981441] debug: unmapping init [mem 0xffffffff91182000-0xffffffff91458fff] [ 4.992334] Write protecting the kernel read-only data: 28672k [ 5.005701] debug: unmapping init [mem 0xffffffff8f803000-0xffffffff8f9fffff] [ 5.009491] debug: unmapping init [mem 0xffffffff90114000-0xffffffff901fffff] [ 5.089194] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.353141] systemd[1]: Detected virtualization kvm. [ 5.355441] systemd[1]: Detected architecture x86-64. [ 5.357717] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.418456] systemd[1]: No hostname configured. [ 5.421798] systemd[1]: Set hostname to . [ 5.425988] random: systemd: uninitialized urandom read (16 bytes read) [ 5.431220] systemd[1]: Initializing machine ID from random generator. [ 5.590527] random: ln: uninitialized urandom read (6 bytes read) [ 5.894979] random: systemd: uninitialized urandom read (16 bytes read) [ 5.898517] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 5.911602] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 5.917423] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Reached target Timers. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. Starting Setup Virtual Console... Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Swap. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 7.210610] device-mapper: uevent: version 1.0.3 [ 7.213524] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 8.950919] virtio_net virtio0 ens2: renamed from eth0 Starting dracut initqueue hook... [ 9.005385] random: fast init done [ 9.318917] scsi host0: ata_piix [ 9.437371] scsi host1: ata_piix [ 9.440537] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 9.444292] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 15.477963] random: crng init done [ 15.481204] random: 7 urandom warning(s) missed due to ratelimiting [ 15.559222] dracut-initqueue[584]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 17.401200] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 19.515395] printk: systemd: 26 output lines suppressed due to ratelimiting [ 20.185600] SELinux: Disabled at runtime. [ 20.281852] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 20.302869] systemd[1]: Detected virtualization kvm. [ 20.306063] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 21.522457] systemd[1]: initrd-switch-root.service: Succeeded. [ 21.526425] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 21.538498] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 21.550403] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 21.555460] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 21.573835] systemd[1]: Starting Journal Service... Starting Journal Service... [ 21.619070] systemd[1]: Listening on Process Core Dump Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Kernel Debug File System... Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-serial\x2dgetty.slice. Mounting POSIX Message Queue File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice User and Session Slice. [ OK ] Listening on udev Control Socket. Mounting Huge Pages File System... [ OK ] Reached target Slices. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Kernel Socket. [ 21.904359] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting udev Coldplug all Devices... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Reached target Paths. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-getty.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Initrd Root File System. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 23.041268] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 24.052452] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 24.184691] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 24.602346] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 24.706267] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit)[ 29.101722] Key type dns_resolver registered [** ] A start job is running for Configur…-only root support (8s / no limit)[ 29.826817] NFS: Registering the id_resolver key type [ 29.832472] Key type id_resolver registered [ 29.842015] Key type id_legacy registered [*** ] A start job is running for Configur…-only root support (8s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Authorization Manager. [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg337-server login: [ 59.945736] libcfs: loading out-of-tree module taints kernel. [ 59.962838] Key type ._llcrypt registered [ 59.964549] Key type .llcrypt registered [ 60.010247] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_hostid [ 68.740830] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 69.382253] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 69.390409] alg: No test for adler32 (adler32-zlib) [ 70.417924] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [ 70.726216] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 72.352240] Key type lgssc registered [ 73.141659] Lustre: Echo OBD driver; http://www.lustre.org/ [ 81.889645] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 101.575764] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 108.344138] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 108.362506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 109.526544] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 109.548452] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 109.609149] Lustre: lustre-MDT0000: new disk, initializing [ 109.668918] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 109.682271] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 111.611870] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 118.748588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 118.798733] Lustre: 6480:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 118.830230] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 118.833479] Lustre: Skipped 1 previous similar message [ 118.880609] Lustre: lustre-MDT0001: new disk, initializing [ 118.910825] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 118.924889] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 118.935544] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 120.704614] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 123.687562] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 128.105907] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 128.242925] Lustre: lustre-OST0000: new disk, initializing [ 128.246021] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 128.271831] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 130.710306] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 132.214486] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 132.218707] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 132.242166] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 137.715127] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 137.786534] Lustre: lustre-OST0001: new disk, initializing [ 137.789156] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 137.819432] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 140.865516] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 140.878087] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 140.905741] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 141.046676] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 148.496561] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 152.780120] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 160.685430] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing check_logdir /tmp/testlogs/ [ 163.006647] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing yml_node [ 165.218498] Lustre: DEBUG MARKER: Client: 2.17.52.1 [ 166.538892] Lustre: DEBUG MARKER: MDS: 2.17.52.1 [ 167.947904] Lustre: DEBUG MARKER: OSS: 2.17.52.1 [ 168.903440] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Apr 10 14:45:29 EDT 2026 [ 178.825458] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 179.684435] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 196.064661] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 196.071313] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 196.084719] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 197.089700] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 197.090529] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 197.098528] Lustre: Skipped 1 previous similar message [ 197.111145] Lustre: Skipped 3 previous similar messages [ 202.208951] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 202.212773] Lustre: Skipped 3 previous similar messages [ 207.328856] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 207.329236] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 207.337345] Lustre: Skipped 3 previous similar messages [ 207.431692] Lustre: server umount lustre-MDT0000 complete [ 209.332831] LustreError: 6473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775846771 with bad export cookie 15297386070509711901 [ 209.337821] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 209.341730] LustreError: 6473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 212.449503] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 212.450293] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 212.458877] Lustre: Skipped 2 previous similar messages [ 215.582297] Lustre: server umount lustre-MDT0001 complete [ 217.630933] Lustre: server umount lustre-OST0000 complete [ 219.788771] Lustre: server umount lustre-OST0001 complete [ 222.689225] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_hostid [ 226.502705] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 242.366390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 242.486606] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 242.505912] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 242.564369] Lustre: lustre-MDT0000: new disk, initializing [ 242.612333] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 242.623489] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 244.228614] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 249.220476] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 249.274385] Lustre: 14943:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 249.458595] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 251.273644] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 255.260577] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 256.845489] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 260.006593] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 260.156204] Lustre: lustre-OST0000: new disk, initializing [ 260.157992] Lustre: Skipped 1 previous similar message [ 260.160490] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 260.163348] Lustre: Skipped 2 previous similar messages [ 260.202431] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 260.207154] Lustre: Skipped 1 previous similar message [ 262.161575] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 262.167591] Lustre: Skipped 1 previous similar message [ 262.170911] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 262.224743] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 262.719302] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 266.604970] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 268.335348] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 268.439885] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 270.105195] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 270.201548] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 272.353560] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 272.355163] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 272.361392] LustreError: Skipped 1 previous similar message [ 272.362586] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 272.368709] Lustre: Skipped 1 previous similar message [ 272.372660] Lustre: Skipped 3 previous similar messages [ 277.232935] Lustre: server umount lustre-OST0000 complete [ 280.033294] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 280.037780] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 292.833558] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 292.836434] Lustre: Skipped 7 previous similar messages [ 293.344428] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 293.426577] Lustre: server umount lustre-MDT0000 complete [ 295.019401] LustreError: 16682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775846857 with bad export cookie 15297386070509713427 [ 295.022118] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 295.025725] LustreError: 16682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 295.165644] Lustre: server umount lustre-MDT0001 complete [ 296.843249] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 14:47:37 (1775846857) [ 297.847920] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 298.516648] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 14:47:39 (1775846859) [ 301.234983] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 301.430584] LustreError: 18944:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 301.441392] LustreError: 18944:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 301.470140] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 303.060252] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 306.657334] LustreError: 18943:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 307.151103] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 309.314244] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 311.974672] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 313.800370] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 316.880639] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 319.628687] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 322.439832] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 336.866390] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 336.871235] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 336.879412] Lustre: Skipped 1 previous similar message [ 336.883859] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 341.003469] Lustre: server umount lustre-OST0000 complete [ 343.530901] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 343.538845] LustreError: Skipped 1 previous similar message [ 349.374970] Lustre: server umount lustre-MDT0000 complete [ 351.433986] LustreError: 18923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775846913 with bad export cookie 15297386070509714316 [ 351.436924] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 351.444018] LustreError: 18923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 351.598860] Lustre: server umount lustre-MDT0001 complete [ 356.939081] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 358.472254] Key type lgssc unregistered [ 358.647168] LNet: 22142:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 358.651790] LNetError: 22142:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 358.667500] LNet: Removed LNI 192.168.203.137@tcp [ 359.090141] Key type .llcrypt unregistered [ 359.092314] Key type ._llcrypt unregistered [ 370.690158] Key type ._llcrypt registered [ 370.692177] Key type .llcrypt registered [ 370.740322] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 371.289337] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 371.298710] alg: No test for adler32 (adler32-zlib) [ 372.185941] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [ 372.292453] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 373.880215] Key type lgssc registered [ 374.405938] Lustre: Echo OBD driver; http://www.lustre.org/ [ 378.587111] loop: module loaded [ 378.620100] loop0: detected capacity change from 0 to 204800000 [ 378.712894] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 378.719576] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 378.892129] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 381.875253] loop0: detected capacity change from 0 to 204800000 [ 381.985652] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 381.991685] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 381.998765] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 384.739475] loop0: detected capacity change from 0 to 204800000 [ 387.585933] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 387.596193] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 388.703130] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 388.723486] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 388.765840] Lustre: lustre-MDT0000: new disk, initializing [ 388.806627] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 388.819119] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 390.663570] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 395.776063] loop1: detected capacity change from 0 to 204800000 [ 398.653952] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 398.819073] Lustre: lustre-OST07c6: new disk, initializing [ 398.822181] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 398.825109] Lustre: Skipped 1 previous similar message [ 398.862720] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 401.709810] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 403.991311] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 403.996242] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 404.019512] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 405.455047] Lustre: 25274:0:(osd_internal.h:1446:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 405.460615] Lustre: 25274:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 405.465104] Lustre: 25274:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 405.470469] Lustre: 25274:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 405.476510] Lustre: 25274:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 405.480348] Lustre: 25274:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 422.122494] Lustre: 24272:0:(osd_internal.h:1446:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 422.127731] Lustre: 24272:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 422.131945] Lustre: 24272:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 422.136216] Lustre: 24272:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/4 [ 422.140881] Lustre: 24272:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 422.145702] Lustre: 24272:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 425.947859] Lustre: 24273:0:(osd_internal.h:1446:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 425.951990] Lustre: 24273:0:(osd_internal.h:1446:osd_trans_exec_op()) Skipped 1 previous similar message [ 425.956311] Lustre: 24273:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 425.961931] Lustre: 24273:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 425.970870] Lustre: 24273:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 425.976259] Lustre: 24273:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 425.985362] Lustre: 24273:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/0, punch: 0/0/0, quota 4/150/0 [ 425.988875] Lustre: 24273:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 425.991704] Lustre: 24273:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 425.996885] Lustre: 24273:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 426.000122] Lustre: 24273:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 426.002524] Lustre: 24273:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 429.078574] Lustre: 24271:0:(osd_internal.h:1446:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 429.082583] Lustre: 24271:0:(osd_internal.h:1446:osd_trans_exec_op()) Skipped 1 previous similar message [ 429.086454] Lustre: 24271:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 429.089732] Lustre: 24271:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 429.093239] Lustre: 24271:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 429.096857] Lustre: 24271:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 429.102309] Lustre: 24271:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/2 [ 429.106663] Lustre: 24271:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 429.110366] Lustre: 24271:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 429.113351] Lustre: 24271:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 429.116454] Lustre: 24271:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 429.119600] Lustre: 24271:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 435.770641] Lustre: 24271:0:(osd_internal.h:1446:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 435.775289] Lustre: 24271:0:(osd_internal.h:1446:osd_trans_exec_op()) Skipped 3 previous similar messages [ 435.782506] Lustre: 24271:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 435.786115] Lustre: 24271:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 435.789965] Lustre: 24271:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 435.794639] Lustre: 24271:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 435.799855] Lustre: 24271:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/2 [ 435.804362] Lustre: 24271:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 435.808054] Lustre: 24271:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 435.811691] Lustre: 24271:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 435.815518] Lustre: 24271:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 435.818989] Lustre: 24271:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 446.179177] Lustre: 24272:0:(osd_internal.h:1446:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 446.184147] Lustre: 24272:0:(osd_internal.h:1446:osd_trans_exec_op()) Skipped 5 previous similar messages [ 446.187698] Lustre: 24272:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 446.191434] Lustre: 24272:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 446.195081] Lustre: 24272:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 446.199099] Lustre: 24272:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 446.202622] Lustre: 24272:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/2 [ 446.206308] Lustre: 24272:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 446.209318] Lustre: 24272:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 446.212225] Lustre: 24272:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 446.215641] Lustre: 24272:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 446.220013] Lustre: 24272:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 489.245926] Lustre: server umount lustre-OST07c6 complete [ 503.332710] Lustre: server umount lustre-MDT0000 complete [ 511.506938] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 512.847701] Key type lgssc unregistered [ 512.998993] LNet: 27787:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 513.003808] LNetError: 27787:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 513.013977] LNet: Removed LNI 192.168.203.137@tcp [ 513.362941] Key type .llcrypt unregistered [ 513.364221] Key type ._llcrypt unregistered [ 522.332521] Key type ._llcrypt registered [ 522.334158] Key type .llcrypt registered [ 522.385398] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 522.922430] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 522.944780] alg: No test for adler32 (adler32-zlib) [ 523.867708] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [ 523.991491] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 525.600144] Key type lgssc registered [ 526.105653] Lustre: Echo OBD driver; http://www.lustre.org/ [ 529.849079] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 529.857767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 531.069739] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 532.890143] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 536.749063] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 536.940733] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 538.698349] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 541.068529] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 542.753970] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 545.878969] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 546.060054] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 548.595987] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 551.057394] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 556.332406] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 562.657132] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 562.660470] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 562.667199] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 565.219031] LustreError: lustre-OST0000-osc-MDT0000: operation ost_setattr to node 0@lo failed: rc = -107 [ 565.219398] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 565.222753] LustreError: Skipped 1 previous similar message [ 565.229147] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 566.757063] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 566.760906] Lustre: Skipped 1 previous similar message [ 568.354301] Lustre: server umount lustre-OST0000 complete [ 571.873881] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 571.875155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 571.883839] Lustre: Skipped 1 previous similar message [ 576.111483] Lustre: server umount lustre-MDT0000 complete [ 576.994161] LustreError: 29019:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 577.003080] LustreError: 29019:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 577.758528] LustreError: 28999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775847140 with bad export cookie 3087401487435880968 [ 577.762437] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 577.767837] LustreError: 28999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 577.897346] Lustre: server umount lustre-MDT0001 complete [ 582.396279] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 583.626986] Key type lgssc unregistered [ 583.773553] LNet: 32396:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 583.777310] LNetError: 32396:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 583.789497] LNet: Removed LNI 192.168.203.137@tcp [ 584.156767] Key type .llcrypt unregistered [ 584.158606] Key type ._llcrypt unregistered [ 592.837956] Key type ._llcrypt registered [ 592.840073] Key type .llcrypt registered [ 592.907950] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 593.271770] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 593.281773] alg: No test for adler32 (adler32-zlib) [ 594.183900] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [ 594.321561] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 595.920265] Key type lgssc registered [ 596.421372] Lustre: Echo OBD driver; http://www.lustre.org/ [ 599.949793] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 599.961091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 601.210263] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 603.163176] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 606.909407] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 607.095352] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 608.831386] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 611.183041] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 612.612627] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 615.254706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 615.391903] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 617.456626] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 619.646451] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 625.645211] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 632.801107] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 632.804756] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 632.810403] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 635.875904] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 635.876436] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 635.883547] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 635.886195] Lustre: Skipped 1 previous similar message [ 640.995785] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 640.998662] Lustre: Skipped 1 previous similar message [ 645.088191] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 645.147158] Lustre: server umount lustre-OST0000 complete [ 648.162620] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 648.166818] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 648.173963] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 651.233748] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 652.636361] Lustre: server umount lustre-MDT0000 complete [ 654.180196] LustreError: 33607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775847216 with bad export cookie 5955892480232731022 [ 654.184106] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 654.184728] LustreError: 33607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 654.344019] Lustre: server umount lustre-MDT0001 complete [ 658.720846] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 660.024455] Key type lgssc unregistered [ 660.172718] LNet: 36825:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 660.176333] LNetError: 36825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 660.184575] LNet: Removed LNI 192.168.203.137@tcp [ 660.549789] Key type .llcrypt unregistered [ 660.551673] Key type ._llcrypt unregistered [ 671.183944] Key type ._llcrypt registered [ 671.185514] Key type .llcrypt registered [ 671.233169] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 671.656926] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 671.676532] alg: No test for adler32 (adler32-zlib) [ 672.544325] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [ 672.641414] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 674.232158] Key type lgssc registered [ 674.668312] Lustre: Echo OBD driver; http://www.lustre.org/ [ 677.486285] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 677.493272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 678.654042] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 680.018474] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 683.116452] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 683.263472] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 684.676670] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 686.640770] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 687.857813] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 690.152416] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 690.266574] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 692.220817] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 694.173184] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 704.615131] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 709.089450] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 709.093708] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 709.099144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 709.790619] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.37@tcp (stopping) [ 710.116038] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 713.189415] Lustre: server umount lustre-MDT0000 complete [ 714.704132] LustreError: 38184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775847277 with bad export cookie 16343600779040389115 [ 714.707479] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 714.709154] LustreError: 38184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 714.897032] LustreError: 38200:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 714.905863] LustreError: 38200:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 714.910424] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.37@tcp (stopping) [ 714.914446] Lustre: Skipped 3 previous similar messages [ 715.234118] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 715.238907] Lustre: Skipped 1 previous similar message [ 719.264895] LustreError: 38200:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 719.265258] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 719.272412] LustreError: 38200:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 719.275282] Lustre: Skipped 1 previous similar message [ 724.449095] LustreError: 39816:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 724.449205] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 724.456215] LustreError: 39816:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 724.462455] Lustre: Skipped 1 previous similar message [ 729.056145] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 729.168056] Lustre: server umount lustre-MDT0001 complete [ 747.356478] Lustre: server umount lustre-OST0000 complete [ 753.302588] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 754.495649] Key type lgssc unregistered [ 754.647524] LNet: 41728:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 754.650481] LNetError: 41728:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 754.658324] LNet: Removed LNI 192.168.203.137@tcp [ 755.002823] Key type .llcrypt unregistered [ 755.004200] Key type ._llcrypt unregistered [ 764.300234] Key type ._llcrypt registered [ 764.301726] Key type .llcrypt registered [ 764.344748] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 764.734449] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 764.759326] alg: No test for adler32 (adler32-zlib) [ 765.614692] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [ 765.694992] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 767.280232] Key type lgssc registered [ 767.673620] Lustre: Echo OBD driver; http://www.lustre.org/ [ 770.457799] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 770.465445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 771.662721] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 773.086175] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 776.188066] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 776.362085] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 777.830526] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 779.835730] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 781.223660] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 783.711417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 783.834435] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 785.870954] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 787.853694] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 797.159804] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 801.248867] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 801.252452] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 801.259252] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 802.273574] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 802.275522] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 802.278534] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 803.482350] Lustre: lustre-OST0000: Not available for connect from 192.168.203.37@tcp (stopping) [ 803.485471] Lustre: Skipped 1 previous similar message [ 807.393981] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 807.398605] Lustre: Skipped 1 previous similar message [ 812.515857] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 812.517977] Lustre: Skipped 2 previous similar messages [ 815.072128] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 815.138436] Lustre: server umount lustre-OST0000 complete [ 828.149935] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 828.150586] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 828.156481] Lustre: Skipped 1 previous similar message [ 828.160626] Lustre: Skipped 2 previous similar messages [ 834.392257] Lustre: server umount lustre-MDT0000 complete [ 835.801418] LustreError: 43086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775847398 with bad export cookie 4085357098468847848 [ 835.805499] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 835.805733] LustreError: 43086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 850.401614] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 850.507407] Lustre: server umount lustre-MDT0001 complete [ 854.420564] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 855.578209] Key type lgssc unregistered [ 855.722466] LNet: 46482:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 855.726316] LNetError: 46482:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 855.735316] LNet: Removed LNI 192.168.203.137@tcp [ 856.065273] Key type .llcrypt unregistered [ 856.066645] Key type ._llcrypt unregistered [ 866.380166] Key type ._llcrypt registered [ 866.381913] Key type .llcrypt registered [ 866.429423] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 866.934251] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 866.945063] alg: No test for adler32 (adler32-zlib) [ 867.812667] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [ 867.907261] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 869.504247] Key type lgssc registered [ 869.916940] Lustre: Echo OBD driver; http://www.lustre.org/ [ 873.094407] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 873.102867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 874.296379] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 875.894751] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 879.203441] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 879.367320] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 880.960203] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 883.186765] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 884.599795] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 887.233160] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 887.380619] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 889.502672] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 891.529124] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 898.665917] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 904.392970] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 905.541165] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 916.380845] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-04-1014h57m57s [ 916.941894] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 919.521945] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 919.523259] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 919.526033] Lustre: Skipped 2 previous similar messages [ 919.528420] Lustre: Skipped 1 previous similar message [ 920.545749] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 920.548439] Lustre: Skipped 2 previous similar messages [ 922.776154] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.37@tcp (stopping) [ 923.951224] Lustre: server umount lustre-MDT0000 complete [ 925.251822] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 925.667793] LustreError: 47854:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 925.677349] LustreError: 47854:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 926.689540] LustreError: 47859:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 926.696971] LustreError: 47859:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 927.813350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 927.872274] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 927.981115] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 928.005672] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 929.000364] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 929.426917] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 930.722257] Lustre: DEBUG MARKER: Wait for df (35956) ... [ 931.390878] Lustre: DEBUG MARKER: done [ 933.345305] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 933.349089] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 933.355913] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 933.358376] Lustre: Skipped 1 previous similar message [ 934.373260] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 934.551447] Lustre: server umount lustre-OST0000 complete [ 939.489925] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 939.489993] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 939.493746] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 939.499020] Lustre: Skipped 1 previous similar message [ 939.503883] Lustre: Skipped 3 previous similar messages [ 942.040660] Lustre: server umount lustre-MDT0000 complete [ 943.485330] LustreError: 47841:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775847506 with bad export cookie 9598542308477193220 [ 943.487670] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 943.490226] LustreError: 47841:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 943.631696] Lustre: server umount lustre-MDT0001 complete [ 948.122825] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 949.356605] Key type lgssc unregistered [ 949.508454] LNet: 52430:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 949.512204] LNetError: 52430:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 949.520711] LNet: Removed LNI 192.168.203.137@tcp [ 949.917288] Key type .llcrypt unregistered [ 949.919271] Key type ._llcrypt unregistered [ 966.111868] Key type ._llcrypt registered [ 966.113379] Key type .llcrypt registered [ 966.166801] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 966.598922] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 966.674562] alg: No test for adler32 (adler32-zlib) [ 967.556522] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [ 967.650991] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 969.248166] Key type lgssc registered [ 969.687236] Lustre: Echo OBD driver; http://www.lustre.org/ [ 972.715317] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 972.722922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 973.829206] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 973.839501] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 973.908968] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 975.460198] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 978.954520] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 979.005356] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 979.019129] Lustre: 54594:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 979.128206] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 980.611229] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 982.641845] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 983.983376] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 986.535519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 986.622764] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 986.705590] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 988.841570] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 990.303239] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 990.872974] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 994.323894] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-04-1014h59m15s [ 994.909276] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 997.495319] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 998.423722] Lustre: DEBUG MARKER: Creating a test file and stat it [ 998.441768] LustreError: 54602:0:(mdt_handler.c:2347:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1018.525261] Lustre: lustre-MDT0000: Client 7e474f4a-5634-4017-a1d8-c10ac5df6dbc (at 192.168.203.37@tcp) reconnecting [ 1038.488130] LustreError: 54602:0:(mdt_handler.c:2347:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1039.396793] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1040.370238] Lustre: DEBUG MARKER: done [ 1043.939598] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 1043.943274] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1043.947835] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1046.496937] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1046.500295] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1046.505855] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1048.237437] Lustre: server umount lustre-OST0000 complete [ 1051.104818] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1051.107525] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1051.112922] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1051.115019] Lustre: Skipped 2 previous similar messages [ 1055.958696] Lustre: server umount lustre-MDT0000 complete [ 1057.262287] LustreError: 54589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775847619 with bad export cookie 60914525814321502 [ 1057.266771] LustreError: 54589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1057.267600] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1057.385685] Lustre: server umount lustre-MDT0001 complete [ 1061.223040] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 1062.352915] Key type lgssc unregistered [ 1062.497686] LNet: 58563:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1062.501875] LNetError: 58563:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1062.511347] LNet: Removed LNI 192.168.203.137@tcp [ 1062.859095] Key type .llcrypt unregistered [ 1062.860991] Key type ._llcrypt unregistered [ 1082.922804] Key type ._llcrypt registered [ 1082.927037] Key type .llcrypt registered [ 1083.000211] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 1083.543748] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1083.621821] alg: No test for adler32 (adler32-zlib) [ 1084.544830] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [ 1084.675674] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 1086.288232] Key type lgssc registered [ 1086.800202] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1088.005963] hrtimer: interrupt took 6377630 ns [ 1088.676464] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1088.694385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1089.886501] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1089.900453] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1089.993403] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1090.536617] Lustre: Failing over lustre-MDT0000 [ 1090.677993] Lustre: server umount lustre-MDT0000 complete [ 1094.471705] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 15:00:55 (1775847655) [ 1097.595047] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1097.827177] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1099.544938] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 1103.288973] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1103.339365] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1103.354817] Lustre: 61319:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1103.474424] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1105.100491] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 1107.219539] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1108.708138] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1111.607479] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1111.700411] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1111.783623] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1114.101646] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 1116.404304] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1119.248040] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1124.860693] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1129.444057] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1129.445335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1129.451168] Lustre: Skipped 2 previous similar messages [ 1129.451462] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1129.454141] Lustre: Skipped 2 previous similar messages [ 1133.059207] Lustre: server umount lustre-MDT0000 complete [ 1135.604845] LustreError: 62052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775847698 with bad export cookie 15710718974951698324 [ 1135.607528] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1135.618272] LustreError: 62052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1135.848065] Lustre: server umount lustre-MDT0001 complete [ 1139.120143] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1143.877493] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1151.252078] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1156.064585] Lustre: 60248:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775847702/real 1775847702] req@000000005925eac8 x1862111222334208/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1775847718 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1156.079974] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1157.088589] Lustre: 60247:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775847703/real 1775847703] req@00000000308718c2 x1862111222334336/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1775847719 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1161.186896] Lustre: 60246:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775847707/real 1775847707] req@00000000bcd09833 x1862111222334464/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1775847723 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1161.447332] LustreError: 62955:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1161.463918] LustreError: 62955:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1161.552577] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1161.640819] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1164.990529] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 1165.543446] LustreError: 62956:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1166.565447] Lustre: 60248:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775847713/real 1775847713] req@000000006ec47c48 x1862111222334848/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1775847729 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1166.582534] LustreError: 64763:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1166.597134] Lustre: 60248:0:(client.c:2479:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1166.622781] LustreError: 64763:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1171.681605] Lustre: 60247:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775847718/real 1775847718] req@0000000000efb2db x1862111222334976/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1775847734 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1171.940777] LustreError: 64764:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1171.977703] LustreError: 64764:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1173.901890] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1174.451905] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1178.680679] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 1179.625994] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1182.761533] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1184.746056] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1185.274453] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1189.860094] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1189.863043] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1189.863323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1189.870574] Lustre: Skipped 2 previous similar messages [ 1194.978253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1194.981361] Lustre: Skipped 4 previous similar messages [ 1195.090278] Lustre: server umount lustre-MDT0000 complete [ 1197.002487] LustreError: 61313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775847759 with bad export cookie 15710718974951700592 [ 1197.006487] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1197.010097] LustreError: 61313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1197.171336] Lustre: server umount lustre-MDT0001 complete [ 1200.721890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1206.247044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1207.266676] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xda07afab33fc3dd3 [ 1207.272869] Lustre: MGC192.168.203.137@tcp: Connection restored to 0@lo (at 0@lo) [ 1207.391529] LustreError: 65484:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1207.444437] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1207.476365] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1209.299997] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 1212.902841] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1213.675176] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1215.968087] Lustre: 60245:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775847762/real 1775847762] req@00000000d4f5ff67 x1862111222357248/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1775847778 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1216.013960] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 1218.652315] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1219.048310] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1220.452480] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1224.161525] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1224.166212] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1224.179352] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1229.959852] Lustre: server umount lustre-MDT0000 complete [ 1230.307025] LustreError: 67220:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1230.317655] LustreError: 67220:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1231.952235] LustreError: 61311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775847794 with bad export cookie 15710718974951701971 [ 1231.957020] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1231.958573] LustreError: 61311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1232.157499] Lustre: server umount lustre-MDT0001 complete [ 1235.531709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1237.420990] Lustre: DEBUG MARKER: files compared the same [ 1247.851417] Lustre: server umount lustre-OST0000 complete [ 1255.596856] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 1256.951087] Key type lgssc unregistered [ 1257.139498] LNet: 70250:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1257.143891] LNetError: 70250:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1257.153458] LNet: Removed LNI 192.168.203.137@tcp [ 1257.518645] Key type .llcrypt unregistered [ 1257.520606] Key type ._llcrypt unregistered [ 1268.883499] Key type ._llcrypt registered [ 1268.885848] Key type .llcrypt registered [ 1268.937400] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 1269.426143] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1269.443954] alg: No test for adler32 (adler32-zlib) [ 1270.352568] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [ 1270.487089] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 1272.080254] Key type lgssc registered [ 1272.573588] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1276.584542] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1276.595516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1277.838609] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1279.522218] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug +malloc all [ 1283.098136] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1283.243612] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1284.961632] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug +malloc all [ 1287.604916] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1289.312202] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1292.507929] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1292.650161] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1295.160942] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug +malloc all [ 1297.450335] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1302.888293] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1313.249492] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1313.249754] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1313.254500] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1313.260331] Lustre: Skipped 1 previous similar message [ 1317.364470] Lustre: server umount lustre-OST0000 complete [ 1319.393301] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1319.400275] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1319.417801] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1319.422268] Lustre: Skipped 1 previous similar message [ 1323.490269] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1323.490987] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1323.503484] Lustre: Skipped 1 previous similar message [ 1325.563421] Lustre: server umount lustre-MDT0000 complete [ 1327.274903] LustreError: 73242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775847889 with bad export cookie 782570130692037468 [ 1327.276564] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1327.282153] LustreError: 73242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1327.419535] Lustre: server umount lustre-MDT0001 complete [ 1332.710479] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 1334.284203] Key type lgssc unregistered [ 1334.476514] LNet: 74825:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1334.483778] LNetError: 74825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1334.494126] LNet: Removed LNI 192.168.203.137@tcp [ 1334.891140] Key type .llcrypt unregistered [ 1334.893165] Key type ._llcrypt unregistered [ 1347.769221] Key type ._llcrypt registered [ 1347.771255] Key type .llcrypt registered [ 1347.830336] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 1348.387683] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1348.399316] alg: No test for adler32 (adler32-zlib) [ 1349.326635] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [ 1349.446518] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 1351.056181] Key type lgssc registered [ 1351.591871] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1355.537812] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1355.548404] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1371.040399] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1376.160253] LustreError: 76177:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.137@tcp: failed processing log, type 4: rc = -110 [ 1402.785211] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1405.086357] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 1407.377841] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1410.649610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1410.726713] Lustre: *** cfs_fail_loc=706, val=0*** [ 1410.896334] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1410.921527] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 1412.630169] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 1416.162063] LustreError: 76203:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1416.173327] LustreError: 76203:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1416.904455] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1417.120794] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1419.000289] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 1421.668427] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1423.417290] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1426.400934] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1426.405165] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1426.412947] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1427.427193] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1427.431331] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1427.437770] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1428.450433] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1428.453081] Lustre: Skipped 1 previous similar message [ 1430.422869] Lustre: server umount lustre-OST0000 complete [ 1432.544983] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1432.549722] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1432.556441] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1438.689330] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1438.696983] Lustre: Skipped 3 previous similar messages [ 1446.368144] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1446.435740] Lustre: server umount lustre-MDT0000 complete [ 1447.907373] LustreError: 76184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775848010 with bad export cookie 6266646880730919902 [ 1447.910093] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1447.912765] LustreError: 76184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1448.042461] Lustre: server umount lustre-MDT0001 complete [ 1452.245564] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 1453.389814] Key type lgssc unregistered [ 1453.528402] LNet: 79438:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1453.533299] LNetError: 79438:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1453.545818] LNet: Removed LNI 192.168.203.137@tcp [ 1453.876502] Key type .llcrypt unregistered [ 1453.878173] Key type ._llcrypt unregistered [ 1465.137120] Key type ._llcrypt registered [ 1465.138640] Key type .llcrypt registered [ 1465.189491] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 1465.624834] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1465.674467] alg: No test for adler32 (adler32-zlib) [ 1466.564153] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [ 1466.681127] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 1468.296924] Key type lgssc registered [ 1468.927955] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1472.873527] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1472.881821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1475.923956] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 1479.437782] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1479.665323] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1481.256669] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 1484.768984] LustreError: 81488:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1484.777782] LustreError: 81488:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1484.936852] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1485.093328] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1487.341723] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 1490.210406] LustreError: 81492:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1490.224573] LustreError: 81492:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1490.909682] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1490.916331] Lustre: MGS: Client 4c09710c-2664-4c83-8054-db7ec26b22ec (at 0@lo) reconnecting [ 1490.926201] LustreError: 81502:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 1491.021145] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1491.054118] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 1492.692947] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 1506.275750] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1506.276686] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1506.281280] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1506.287611] Lustre: Skipped 1 previous similar message [ 1506.295193] Lustre: Skipped 1 previous similar message [ 1508.322542] Lustre: server umount lustre-OST0000 complete [ 1511.393452] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1511.394050] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1511.397687] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1511.402523] Lustre: Skipped 1 previous similar message [ 1516.512938] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1516.514907] Lustre: Skipped 2 previous similar messages [ 1521.632957] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1521.635692] Lustre: Skipped 1 previous similar message [ 1524.192191] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1524.238765] Lustre: server umount lustre-MDT0000 complete [ 1525.729876] LustreError: 81488:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1525.738036] LustreError: 81488:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1525.935853] Lustre: server umount lustre-MDT0001 complete [ 1527.496941] Lustre: server umount MGS complete [ 1532.110295] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 15:08:13 (1775848093) [ 1543.520189] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_hostid [ 1546.996690] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 1560.123344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1561.631190] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 1565.864407] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1565.949725] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 1565.965701] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1566.000378] Lustre: lustre-MDT0001: new disk, initializing [ 1566.025444] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1566.034481] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1566.038627] Lustre: 88549:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 1566.043544] LustreError: 88549:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 1567.416434] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 1572.213636] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1572.319352] Lustre: lustre-OST0000: new disk, initializing [ 1572.322622] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1572.355895] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1574.625969] Lustre: 89468:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@000000008daafcda x1862111622892672/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1775848153 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1574.780587] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 1578.460061] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1582.343699] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1582.354820] Lustre: MGS: Client 094e026f-5c5b-4ab7-bcdf-e7fc0dc4c13f (at 0@lo) reconnecting [ 1582.384449] Lustre: 87836:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1582.395128] LustreError: 88535:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 1582.400667] LustreError: 88535:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 1582.407076] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1582.457742] Lustre: lustre-MDT0000: new disk, initializing [ 1582.577708] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1582.592597] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1583.659583] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1583.674105] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1583.763461] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1587.380920] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 1618.405635] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1618.426248] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1618.436240] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1629.672857] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1629.965278] Lustre: Skipped 4 previous similar messages [ 1631.200154] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1631.729171] Lustre: server umount lustre-OST0000 complete [ 1639.036862] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1639.046013] Lustre: Skipped 1 previous similar message [ 1646.684796] Lustre: server umount lustre-MDT0000 complete [ 1649.633689] LustreError: 88520:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1649.684739] LustreError: 88520:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1660.400587] Lustre: server umount lustre-MDT0001 complete [ 1664.227270] Lustre: server umount MGS complete [ 1674.769694] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 15:10:34 (1775848234) [ 1691.133859] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 1693.572626] Key type lgssc unregistered [ 1693.857396] LNet: 92724:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1693.872347] LNetError: 92724:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1693.891979] LNet: Removed LNI 192.168.203.137@tcp [ 1694.663794] Key type .llcrypt unregistered [ 1694.666326] Key type ._llcrypt unregistered [ 1709.025308] Key type ._llcrypt registered [ 1709.027606] Key type .llcrypt registered [ 1709.121472] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 1709.955074] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1710.068895] alg: No test for adler32 (adler32-zlib) [ 1711.220271] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [ 1711.458731] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 1713.144235] Key type lgssc registered [ 1714.162209] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1720.769383] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1720.788734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1722.269336] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1725.764389] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 1731.567821] LustreError: 93917:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1731.589910] LustreError: 93917:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1733.823865] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1734.061503] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1738.120842] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 1743.422458] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1747.057283] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1753.300296] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1753.676922] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1759.240316] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 1764.922371] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1767.028948] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 1777.636630] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1777.645833] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1777.647853] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1777.662216] Lustre: Skipped 1 previous similar message [ 1782.769782] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1782.773425] Lustre: Skipped 1 previous similar message [ 1783.228902] Lustre: server umount lustre-OST0000 complete [ 1787.893565] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1787.904779] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1787.919891] Lustre: Skipped 1 previous similar message [ 1787.933823] Lustre: Skipped 2 previous similar messages [ 1793.005452] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1793.026404] Lustre: Skipped 1 previous similar message [ 1798.116968] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1798.135107] Lustre: Skipped 1 previous similar message [ 1801.184121] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1802.985856] Lustre: server umount lustre-MDT0000 complete [ 1803.491304] LustreError: 93917:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1803.522901] LustreError: 93917:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1807.588518] LustreError: 96074:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1807.628954] LustreError: 96074:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1810.044089] LustreError: 93897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775848372 with bad export cookie 8623474484774843579 [ 1810.073875] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1810.099838] LustreError: 93897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1811.722738] Lustre: server umount lustre-MDT0001 complete [ 1829.445697] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 1832.637466] Key type lgssc unregistered [ 1832.948981] LNet: 97136:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1832.958944] LNetError: 97136:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1832.997971] LNet: Removed LNI 192.168.203.137@tcp [ 1833.769338] Key type .llcrypt unregistered [ 1833.772849] Key type ._llcrypt unregistered [ 1849.817465] Key type ._llcrypt registered [ 1849.819798] Key type .llcrypt registered [ 1849.916592] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 1851.089333] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1851.138388] alg: No test for adler32 (adler32-zlib) [ 1852.179365] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [ 1852.509670] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 1854.312195] Key type lgssc registered [ 1855.209920] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1861.928318] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1861.932806] LustreError: 98198:0:(tgt_mount.c:2501:server_fill_super()) cfs_race id 716 sleeping [ 1861.976449] LustreError: 98205:0:(tgt_mount.c:2501:server_fill_super()) cfs_fail_race id 716 waking [ 1861.982841] LustreError: 98198:0:(tgt_mount.c:2501:server_fill_super()) cfs_fail_race id 716 awake: rc=4958 [ 1861.988377] LustreError: 98198:0:(tgt_mount.c:2513:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 1862.001206] LustreError: 98198:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1862.001791] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1863.349574] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1872.145445] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1872.457418] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1877.909197] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 1883.166555] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing lsmod [ 1886.395241] LustreError: 99379:0:(tgt_mount.c:2501:server_fill_super()) cfs_race id 716 sleeping [ 1886.417628] LustreError: 99380:0:(tgt_mount.c:2501:server_fill_super()) cfs_fail_race id 716 waking [ 1886.426270] LustreError: 99379:0:(tgt_mount.c:2501:server_fill_super()) cfs_fail_race id 716 awake: rc=4978 [ 1886.433816] LustreError: 99379:0:(tgt_mount.c:2513:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 1886.445569] LustreError: 99379:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1886.451946] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1886.876936] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1887.915757] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:65) [ 1892.833835] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1892.846939] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1892.869332] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1893.353243] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1893.363016] Lustre: Skipped 1 previous similar message [ 1896.058071] Lustre: server umount lustre-MDT0000 complete [ 1897.444041] LustreError: 98243:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1897.475894] LustreError: 98243:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1900.212227] LustreError: 98220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775848462 with bad export cookie 18441913125862964791 [ 1900.230379] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1900.238343] Lustre: 98235:0:(mgc_request.c:1917:mgc_process_log()) MGC192.168.203.137@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 1901.601712] LustreError: 98675:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000005ce59f34 x1862112027628160/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1901.976940] Lustre: server umount lustre-MDT0001 complete [ 1917.424689] Lustre: server umount lustre-OST0000 complete [ 1927.816499] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1928.601715] LustreError: 100443:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1928.745902] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1933.809249] LustreError: 100444:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1934.979676] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 1938.921622] LustreError: 100443:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1944.043645] LustreError: 100444:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1945.192154] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1945.759729] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1950.807928] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 1957.057104] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1961.421371] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1969.511647] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1969.890884] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1974.049437] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:97) [ 1976.719876] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 1982.679173] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1991.655372] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1991.671782] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1991.698843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1991.705802] Lustre: Skipped 3 previous similar messages [ 1991.876668] Lustre: server umount lustre-OST0000 complete [ 1996.769486] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1996.788129] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1996.803858] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1999.842985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1999.863603] Lustre: Skipped 1 previous similar message [ 2004.962359] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2004.975723] Lustre: Skipped 1 previous similar message [ 2006.107723] Lustre: server umount lustre-MDT0000 complete [ 2010.083283] LustreError: 101413:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2010.110559] LustreError: 101413:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2010.357337] LustreError: 100425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775848572 with bad export cookie 18441913125862965715 [ 2010.362094] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2010.373237] LustreError: 100425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2010.860915] Lustre: server umount lustre-MDT0001 complete [ 2024.476931] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 2027.708794] Key type lgssc unregistered [ 2028.042551] LNet: 103625:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2028.051722] LNetError: 103625:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2028.072694] LNet: Removed LNI 192.168.203.137@tcp [ 2028.972615] Key type .llcrypt unregistered [ 2028.982153] Key type ._llcrypt unregistered [ 2058.227715] Key type ._llcrypt registered [ 2058.231503] Key type .llcrypt registered [ 2058.462886] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 2059.985370] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2060.018392] alg: No test for adler32 (adler32-zlib) [ 2061.097113] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [ 2061.350296] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 2063.090815] Key type lgssc registered [ 2064.229951] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2072.534502] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2072.584902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2074.278579] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2078.492873] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 2083.818505] LustreError: 105027:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2083.851608] LustreError: 105027:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2088.247943] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2088.767708] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2094.326617] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 2102.569867] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2107.090738] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2116.587318] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2117.166575] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2119.234024] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 2124.504603] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 2131.630869] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2140.090683] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 2144.132835] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 2149.867950] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2149.882341] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2149.897976] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2153.965543] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2153.968882] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2153.992583] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2153.996731] Lustre: Skipped 1 previous similar message [ 2155.765383] Lustre: server umount lustre-OST0000 complete [ 2160.610828] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2160.617254] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2160.628917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2164.195815] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2164.197416] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2164.227562] Lustre: Skipped 1 previous similar message [ 2166.374488] Lustre: server umount lustre-MDT0000 complete [ 2169.316411] LustreError: 107246:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2169.336334] LustreError: 107246:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2170.214432] LustreError: 105744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775848732 with bad export cookie 4923078808414581842 [ 2170.218789] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2170.223086] LustreError: 105744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2170.495758] Lustre: server umount lustre-MDT0001 complete [ 2183.681366] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 2187.991404] Key type lgssc unregistered [ 2188.398642] LNet: 108313:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2188.410394] LNetError: 108313:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2189.484554] LNet: Removed LNI 192.168.203.137@tcp [ 2190.666514] Key type .llcrypt unregistered [ 2190.669215] Key type ._llcrypt unregistered [ 2212.083578] Key type ._llcrypt registered [ 2212.089773] Key type .llcrypt registered [ 2212.243324] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 2213.084051] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2213.121345] alg: No test for adler32 (adler32-zlib) [ 2214.358986] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [ 2214.709623] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 2216.464179] Key type lgssc registered [ 2218.024383] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2227.227499] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2227.244575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2228.967917] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2233.847732] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 2239.457681] LustreError: 109518:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2239.484182] LustreError: 109518:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2243.676537] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2244.004194] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2249.042149] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 2255.851305] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2261.025392] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2269.334203] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2269.844344] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2273.974125] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 2277.442925] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 2284.272115] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2299.875418] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2299.887951] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2299.889700] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2299.903248] Lustre: Skipped 1 previous similar message [ 2299.919140] Lustre: Skipped 1 previous similar message [ 2304.335356] Lustre: server umount lustre-OST0000 complete [ 2310.116070] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2310.120344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2310.140301] Lustre: Skipped 1 previous similar message [ 2310.150202] Lustre: Skipped 1 previous similar message [ 2314.833336] Lustre: server umount lustre-MDT0000 complete [ 2315.235583] LustreError: 111299:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2315.267212] LustreError: 111299:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2318.544048] LustreError: 109497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775848881 with bad export cookie 18195690885171026193 [ 2318.545200] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2318.555923] LustreError: 109497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2318.974514] Lustre: server umount lustre-MDT0001 complete [ 2330.480026] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 2333.541588] Key type lgssc unregistered [ 2333.825873] LNet: 112720:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2333.835310] LNetError: 112720:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2333.862853] LNet: Removed LNI 192.168.203.137@tcp [ 2334.841071] Key type .llcrypt unregistered [ 2334.848085] Key type ._llcrypt unregistered [ 2360.679975] Key type ._llcrypt registered [ 2360.682757] Key type .llcrypt registered [ 2360.764334] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 2362.243851] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2362.316657] alg: No test for adler32 (adler32-zlib) [ 2363.443491] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [ 2363.708050] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 2365.408180] Key type lgssc registered [ 2366.528239] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2374.628666] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2374.645545] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2376.176277] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2381.133844] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 2386.402762] LustreError: 114121:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2386.425255] LustreError: 114121:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2390.456409] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2390.956819] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2395.440499] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 2401.589978] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2405.928484] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2413.823647] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2414.356846] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2421.503137] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 2421.719670] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 2428.332545] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2436.699669] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 2440.951583] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 2440.956991] Lustre: Skipped 2 previous similar messages [ 2444.917533] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 2444.987318] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 2446.461609] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 2446.467473] Lustre: Skipped 2 previous similar messages [ 2452.081596] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2452.095872] Lustre: Skipped 1 previous similar message [ 2457.531276] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.37@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 2457.540428] Lustre: Skipped 2 previous similar messages [ 2459.903855] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.37@tcp 192.168.0.[2,10]@tcp [ 2468.359915] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.37/24@tcp in log lustre-MDT0000 [ 2468.370757] Lustre: Skipped 2 previous similar messages [ 2475.119900] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.37/24@tcp [ 2475.127366] Lustre: Skipped 1 previous similar message [ 2483.172711] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2483.177394] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2483.179963] LustreError: Skipped 1 previous similar message [ 2483.180402] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2483.216953] Lustre: Skipped 1 previous similar message [ 2488.297587] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2488.304717] Lustre: Skipped 2 previous similar messages [ 2488.524082] Lustre: server umount lustre-OST0000 complete [ 2493.408893] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2493.409042] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2493.416790] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2493.437914] Lustre: Skipped 1 previous similar message [ 2498.529751] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2498.545737] Lustre: Skipped 2 previous similar messages [ 2499.864312] Lustre: server umount lustre-MDT0000 complete [ 2503.659021] LustreError: 116226:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2503.678642] LustreError: 116226:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2504.164678] LustreError: 114102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775849066 with bad export cookie 10138565326336899595 [ 2504.174246] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2504.663151] Lustre: server umount lustre-MDT0001 complete [ 2518.862790] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 2523.010690] Key type lgssc unregistered [ 2523.414295] LNet: 118618:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2523.438673] LNetError: 118618:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2523.483398] LNet: Removed LNI 192.168.203.137@tcp [ 2524.494793] Key type .llcrypt unregistered [ 2524.496537] Key type ._llcrypt unregistered [ 2555.240878] Key type ._llcrypt registered [ 2555.242934] Key type .llcrypt registered [ 2555.339059] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 2556.584226] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2556.628558] alg: No test for adler32 (adler32-zlib) [ 2557.875402] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [ 2558.094817] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 2559.760338] Key type lgssc registered [ 2561.020635] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2570.301148] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2570.323404] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2571.905190] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 2571.956234] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.37/24@tcp [ 2571.968533] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2576.569455] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 2581.479085] LustreError: 120163:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2581.499994] LustreError: 120163:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2586.562908] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2586.594620] LustreError: 120164:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2587.176570] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2587.195909] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.37/24@tcp [ 2587.264471] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2592.051335] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 2598.508888] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2602.556815] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2610.117736] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2610.512835] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2616.765306] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:201 to 0x280000401:225) [ 2617.021158] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 2623.025361] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2637.281842] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2637.288975] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2637.319776] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2639.995941] Lustre: server umount lustre-OST0000 complete [ 2647.521873] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2647.531937] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2647.559149] Lustre: Skipped 2 previous similar messages [ 2647.581506] Lustre: Skipped 2 previous similar messages [ 2652.648442] LustreError: 120159:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2652.682128] LustreError: 120159:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2653.245822] Lustre: server umount lustre-MDT0000 complete [ 2656.737676] LustreError: 120163:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2656.751174] LustreError: 120163:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2657.785588] LustreError: 120144:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775849220 with bad export cookie 16366821146690055335 [ 2657.790345] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2657.797408] LustreError: 120144:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2658.327843] Lustre: server umount lustre-MDT0001 complete [ 2671.280878] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 2674.781650] Key type lgssc unregistered [ 2675.246205] LNet: 123400:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2675.256431] LNetError: 123400:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2675.301973] LNet: Removed LNI 192.168.203.137@tcp [ 2676.390316] Key type .llcrypt unregistered [ 2676.392277] Key type ._llcrypt unregistered [ 2710.207153] Key type ._llcrypt registered [ 2710.209050] Key type .llcrypt registered [ 2710.293599] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_hostid [ 2727.522511] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 2728.940751] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2728.989968] alg: No test for adler32 (adler32-zlib) [ 2730.033819] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [ 2730.429157] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 2732.225830] Key type lgssc registered [ 2733.586249] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2773.671302] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2773.695187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2774.963824] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2775.029057] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2775.126971] Lustre: lustre-MDT0000: new disk, initializing [ 2775.253661] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2775.286185] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2779.677804] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 2791.080069] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2791.245671] Lustre: 126770:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2791.371716] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2791.378964] Lustre: Skipped 1 previous similar message [ 2791.485243] Lustre: lustre-MDT0001: new disk, initializing [ 2791.610397] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2791.673355] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2791.689374] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2796.442706] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 2804.720583] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2809.514929] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2817.278826] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2817.625518] Lustre: lustre-OST0000: new disk, initializing [ 2817.631714] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2817.747676] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2823.965525] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 2826.315777] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2826.325910] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2826.444040] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2832.071756] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2837.141604] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2837.407981] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2842.658192] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2842.903976] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2847.213809] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2847.217111] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2847.217865] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2847.230995] Lustre: Skipped 1 previous similar message [ 2851.301569] Lustre: server umount lustre-OST0000 complete [ 2857.444742] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2857.448233] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2857.466497] Lustre: Skipped 1 previous similar message [ 2857.473412] Lustre: Skipped 2 previous similar messages [ 2861.640896] Lustre: server umount lustre-MDT0000 complete [ 2865.241738] LustreError: 126762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775849427 with bad export cookie 7733680428770241371 [ 2865.256309] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2865.619708] Lustre: server umount lustre-MDT0001 complete [ 2873.969432] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2874.736775] LustreError: 130580:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2874.777645] LustreError: 130580:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2874.887583] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2879.372376] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 2879.971510] LustreError: 130581:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2885.116860] LustreError: 130580:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2888.482746] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2888.978545] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2893.276561] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 2899.489429] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2903.629464] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2910.857512] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2911.193329] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2918.313316] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 2924.469290] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2929.406156] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2929.639529] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2934.853425] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2935.087663] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2944.745993] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2944.944217] Lustre: lustre-OST0001: new disk, initializing [ 2944.948710] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2945.053612] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2951.826979] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 2952.323973] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2952.331272] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2952.443613] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2961.226265] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2961.470446] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2966.342460] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2966.585824] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2971.173110] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8a2b06b27800.ost_server_uuid 50 [ 2973.044123] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8a2b06b27800.ost_server_uuid in IDLE state after 0 sec [ 2983.564350] Lustre: server umount lustre-OST0001 complete [ 2988.522887] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2988.528627] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2988.530326] LustreError: Skipped 2 previous similar messages [ 2988.532097] LustreError: 133237:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2988.584724] Lustre: Skipped 1 previous similar message [ 2988.603514] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2988.614610] Lustre: Skipped 1 previous similar message [ 2993.638703] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2993.642188] LustreError: 133237:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2993.646264] Lustre: Skipped 1 previous similar message [ 2993.667643] LustreError: 133237:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2998.757332] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2998.774688] Lustre: Skipped 1 previous similar message [ 3002.339829] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3002.623807] Lustre: server umount lustre-OST0000 complete [ 3009.018756] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3009.037733] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3009.041037] Lustre: Skipped 3 previous similar messages [ 3009.072033] Lustre: Skipped 1 previous similar message [ 3013.584422] Lustre: server umount lustre-MDT0000 complete [ 3014.114741] LustreError: 132198:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3014.142788] LustreError: 132198:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3017.425524] LustreError: 130560:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775849579 with bad export cookie 7733680428770242253 [ 3017.441864] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3017.444138] LustreError: 130560:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 3017.922899] Lustre: server umount lustre-MDT0001 complete [ 3040.470894] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 3043.501896] Key type lgssc unregistered [ 3043.826102] LNet: 136422:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3043.840957] LNetError: 136422:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3043.864595] LNet: Removed LNI 192.168.203.137@tcp [ 3044.765508] Key type .llcrypt unregistered [ 3044.768451] Key type ._llcrypt unregistered [ 3090.240750] Key type ._llcrypt registered [ 3090.243790] Key type .llcrypt registered [ 3090.347808] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_hostid [ 3105.745106] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 3106.806616] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3106.972232] alg: No test for adler32 (adler32-zlib) [ 3108.225192] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [ 3108.622024] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 3110.384186] Key type lgssc registered [ 3111.569469] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3149.106405] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3149.212971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3150.450850] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3150.488732] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3150.593233] Lustre: lustre-MDT0000: new disk, initializing [ 3150.664654] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3150.679787] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3154.917745] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 3165.806512] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3165.936022] Lustre: 140544:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3165.999628] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3166.003922] Lustre: Skipped 1 previous similar message [ 3166.155808] Lustre: lustre-MDT0001: new disk, initializing [ 3166.277754] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3166.331917] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3166.351911] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3170.782925] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 3178.926480] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3183.305868] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3191.088302] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3191.473575] Lustre: lustre-OST0000: new disk, initializing [ 3191.482142] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3191.637820] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3199.261716] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 3201.117703] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3201.135706] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3201.256711] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3207.471779] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3213.154797] Lustre: Failing over lustre-OST0000 [ 3213.498771] Lustre: server umount lustre-OST0000 complete [ 3231.206151] Lustre: 138771:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775849777/real 1775849777] req@00000000ba90c964 x1862113344651008/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1775849793 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3231.242283] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3232.736701] Lustre: 138768:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775849779/real 1775849779] req@00000000b7476b93 x1862113344651136/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1775849795 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3232.767576] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3233.024706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3233.355265] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3233.368836] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3234.338919] Lustre: 138770:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775849779/real 1775849779] req@000000007aad99de x1862113344652032/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1775849795 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3238.442793] Lustre: 138771:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775849784/real 1775849784] req@000000002d058dec x1862113344652928/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1775849800 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3238.460348] Lustre: 138771:0:(client.c:2479:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3238.946560] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 3240.501960] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3242.529096] Lustre: 138770:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775849789/real 1775849789] req@00000000b7417db5 x1862113344653952/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1775849805 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3242.554215] Lustre: 138770:0:(client.c:2479:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3242.601889] Lustre: Failing over lustre-MDT0000 [ 3242.610161] LustreError: 138767:0:(import.c:706:ptlrpc_connect_import_locked()) can't connect to a closed import [ 3242.640984] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3242.643069] Lustre: lustre-OST0000: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [ 3242.825300] Lustre: server umount lustre-MDT0000 complete [ 3242.978379] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3242.986115] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3245.603331] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3245.617197] Lustre: Skipped 1 previous similar message [ 3250.660430] LustreError: 141587:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3250.690022] LustreError: 141587:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 3252.370303] LustreError: 140553:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3255.791221] LustreError: 140551:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3255.818557] LustreError: 140551:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3260.900702] LustreError: 140556:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3260.937148] LustreError: 140556:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3261.168498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3261.435579] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3261.631659] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 3261.698897] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3261.760171] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3262.625554] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3266.277898] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 3267.077827] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3267.105991] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 3267.174449] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 3277.295419] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3277.301090] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3277.302242] Lustre: Skipped 1 previous similar message [ 3277.323521] Lustre: Skipped 1 previous similar message [ 3281.370961] Lustre: server umount lustre-OST0000 complete [ 3287.526416] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3287.533815] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3287.534559] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3287.539904] Lustre: Skipped 1 previous similar message [ 3291.364431] Lustre: server umount lustre-MDT0000 complete [ 3292.641111] LustreError: 140556:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3292.665746] LustreError: 140556:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3294.931964] LustreError: 142286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775849857 with bad export cookie 18205698523411403027 [ 3294.934190] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3294.946260] LustreError: 142286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3295.461823] Lustre: server umount lustre-MDT0001 complete [ 3305.347207] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 3308.015920] Key type lgssc unregistered [ 3308.256235] LNet: 145787:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3308.265900] LNetError: 145787:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3308.279854] LNet: Removed LNI 192.168.203.137@tcp [ 3309.073849] Key type .llcrypt unregistered [ 3309.079993] Key type ._llcrypt unregistered [ 3345.559918] Key type ._llcrypt registered [ 3345.563914] Key type .llcrypt registered [ 3345.643546] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 3346.909413] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3346.981637] alg: No test for adler32 (adler32-zlib) [ 3348.116606] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [ 3348.415915] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 3350.064647] Key type lgssc registered [ 3351.400471] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3361.356296] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3361.384402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3363.129110] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3368.864354] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 3372.516460] LustreError: 147291:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3372.552066] LustreError: 147291:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3377.355213] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3377.634662] LustreError: 147290:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3381.583323] LustreError: 147319:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3381.593275] LustreError: 147319:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3381.612029] LustreError: 147319:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 0, failed: rc = -5 [ 3381.994323] Lustre: server umount lustre-MDT0000 complete [ 3397.000602] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_hostid [ 3407.711139] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 3447.990895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3448.276050] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3448.309611] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3448.436910] Lustre: lustre-MDT0000: new disk, initializing [ 3448.498201] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3448.512345] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3452.532303] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 3463.373731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3463.523185] Lustre: 150479:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3463.579517] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3463.589911] Lustre: Skipped 1 previous similar message [ 3463.693950] Lustre: lustre-MDT0001: new disk, initializing [ 3463.790840] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3463.841400] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3463.862541] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3468.346538] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 3477.090696] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3481.970418] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3489.367211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3489.745337] Lustre: lustre-OST0000: new disk, initializing [ 3489.749260] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3489.859953] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3495.517404] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3495.532650] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3495.758602] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3496.472490] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 3503.952455] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3508.446257] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 3510.143986] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 15:41:10 (1775850070) [ 3797.987059] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3797.991822] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3798.003039] Lustre: Skipped 2 previous similar messages [ 3798.025831] Lustre: Skipped 2 previous similar messages [ 3802.916692] Lustre: server umount lustre-MDT0000 complete [ 3803.107471] LustreError: 153884:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3803.153840] LustreError: 153884:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3807.726978] LustreError: 150471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775850370 with bad export cookie 2155350806848299944 [ 3807.730822] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3807.739403] LustreError: 150471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3808.224911] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3808.235649] LustreError: 150485:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3808.256993] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3812.256883] LustreError: 150486:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3812.259382] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3817.445454] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3822.048141] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3822.216104] Lustre: server umount lustre-MDT0001 complete [ 3832.589858] Lustre: server umount lustre-OST0000 complete [ 3842.526525] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_hostid [ 3853.112347] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 3893.251747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3893.494559] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3893.527646] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3893.608373] Lustre: lustre-MDT0000: new disk, initializing [ 3893.688852] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3893.707752] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3898.500266] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 3910.058868] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3910.186313] Lustre: 156744:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3910.284345] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3910.294921] Lustre: Skipped 1 previous similar message [ 3910.326090] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3910.329084] Lustre: Skipped 1 previous similar message [ 3910.392250] Lustre: lustre-MDT0001: new disk, initializing [ 3910.503796] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3910.543251] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3910.563810] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3915.386303] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 3923.969804] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3928.395975] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3936.437868] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3936.765855] Lustre: lustre-OST0000: new disk, initializing [ 3936.771855] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3936.899743] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3944.189432] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 3947.112315] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3947.131249] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3947.307434] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3952.597468] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3957.993182] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3958.236329] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3962.560653] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3962.843402] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3967.972418] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3967.975585] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3967.987819] Lustre: Skipped 1 previous similar message [ 3967.988847] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3968.017503] Lustre: Skipped 1 previous similar message [ 3971.307683] Lustre: server umount lustre-OST0000 complete [ 3977.185110] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3977.198251] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3982.100329] Lustre: server umount lustre-MDT0000 complete [ 3983.333809] LustreError: 156755:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3983.356662] LustreError: 156755:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3986.252115] LustreError: 158486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775850548 with bad export cookie 2155350806848458774 [ 3986.263254] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3986.271780] LustreError: 158486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3986.580637] Lustre: server umount lustre-MDT0001 complete [ 3995.888348] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 15:49:16 (1775850556) [ 4012.415942] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_hostid [ 4023.732937] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 4065.802039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4066.334952] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4066.371366] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4066.496845] Lustre: lustre-MDT0000: new disk, initializing [ 4066.610360] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4066.638684] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4071.767567] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 4084.164354] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4084.281883] Lustre: 163060:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4084.596769] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4089.266742] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 4099.161078] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4104.138840] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4112.480302] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4112.918915] Lustre: lustre-OST0000: new disk, initializing [ 4112.927017] Lustre: Skipped 1 previous similar message [ 4112.943781] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4112.950607] Lustre: Skipped 2 previous similar messages [ 4120.346898] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 4120.721792] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4120.735755] Lustre: Skipped 1 previous similar message [ 4120.750607] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4120.972910] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4129.133932] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4141.539164] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4141.542046] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4141.542366] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4141.542370] Lustre: Skipped 3 previous similar messages [ 4141.562529] Lustre: Skipped 2 previous similar messages [ 4144.136960] Lustre: server umount lustre-OST0000 complete [ 4151.266062] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4151.276860] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4154.373304] Lustre: server umount lustre-MDT0000 complete [ 4156.901994] LustreError: 163068:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4156.929318] LustreError: 163068:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4158.712174] LustreError: 163054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775850721 with bad export cookie 2155350806848459656 [ 4158.714198] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4158.728692] LustreError: 163054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4159.109869] Lustre: server umount lustre-MDT0001 complete [ 4168.491788] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 15:52:08 (1775850728) [ 4187.197691] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_hostid [ 4198.393849] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 4241.348663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4241.653621] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4241.739596] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4241.911328] Lustre: lustre-MDT0000: new disk, initializing [ 4242.082475] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4242.095431] Lustre: Skipped 2 previous similar messages [ 4242.155877] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4247.236881] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 4259.647967] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4259.806945] Lustre: 168993:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4260.168495] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4266.559623] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 4275.594788] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4280.196123] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4287.596051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4293.873217] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 4294.708726] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4294.845952] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4301.768191] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4310.498960] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4310.502628] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4310.507057] LustreError: Skipped 1 previous similar message [ 4310.508098] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4310.508103] Lustre: Skipped 3 previous similar messages [ 4310.536727] Lustre: Skipped 2 previous similar messages [ 4316.544300] Lustre: server umount lustre-OST0000 complete [ 4326.730715] Lustre: server umount lustre-MDT0000 complete [ 4330.992853] LustreError: 169005:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4331.007249] LustreError: 169005:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4331.100619] LustreError: 168985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775850893 with bad export cookie 2155350806848460741 [ 4331.109161] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4331.115724] LustreError: 168985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4331.572538] Lustre: server umount lustre-MDT0001 complete [ 4344.022755] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 4347.245027] Key type lgssc unregistered [ 4347.507045] LNet: 172469:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4347.518249] LNetError: 172469:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4347.532831] LNet: Removed LNI 192.168.203.137@tcp [ 4348.344125] Key type .llcrypt unregistered [ 4348.348243] Key type ._llcrypt unregistered [ 4372.270813] Key type ._llcrypt registered [ 4372.274663] Key type .llcrypt registered [ 4372.409697] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 4373.597118] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4373.652271] alg: No test for adler32 (adler32-zlib) [ 4374.699743] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [ 4374.983558] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 4376.784357] Key type lgssc registered [ 4378.151930] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4386.917615] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4386.977753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4388.583314] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4393.134672] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 4399.090318] LustreError: 173879:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4399.105822] LustreError: 173879:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4403.124043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4403.557601] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4407.911518] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 4414.029619] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4418.013312] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4425.622474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4426.028211] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4432.135196] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 4434.241249] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4437.987476] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4458.978391] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4458.986581] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4459.008484] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4460.006728] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4460.007278] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4460.019801] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4462.917322] Lustre: server umount lustre-OST0000 complete [ 4470.243880] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4470.246259] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4470.261157] Lustre: Skipped 1 previous similar message [ 4470.278574] Lustre: Skipped 2 previous similar messages [ 4473.452235] Lustre: server umount lustre-MDT0000 complete [ 4475.373852] LustreError: 173873:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4475.392926] LustreError: 173873:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4477.516234] LustreError: 173858:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775851040 with bad export cookie 15300065864923200145 [ 4477.537146] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4478.085938] Lustre: server umount lustre-MDT0001 complete [ 4488.793136] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 4491.798530] Key type lgssc unregistered [ 4492.057824] LNet: 177082:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4492.069361] LNetError: 177082:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4492.093398] LNet: Removed LNI 192.168.203.137@tcp [ 4492.991798] Key type .llcrypt unregistered [ 4492.996612] Key type ._llcrypt unregistered [ 4517.791482] Key type ._llcrypt registered [ 4517.793724] Key type .llcrypt registered [ 4517.890448] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 4518.907888] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4518.939885] alg: No test for adler32 (adler32-zlib) [ 4520.041648] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [ 4520.304715] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 4521.984324] Key type lgssc registered [ 4523.065663] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4530.944326] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4530.965659] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4532.707258] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4537.418398] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 4542.964290] LustreError: 178489:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4542.999668] LustreError: 178489:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4546.741169] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4547.367887] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4551.948322] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 4557.106103] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4561.402727] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4568.366608] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4568.850158] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4575.607940] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 4578.107519] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 4582.472794] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4590.284421] Lustre: lustre-OST0000: Not available for connect from 192.168.203.37@tcp (stopping) [ 4592.610552] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4592.619067] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4592.672658] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4593.641506] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4593.660304] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4593.683309] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4593.694355] Lustre: Skipped 1 previous similar message [ 4595.636927] Lustre: server umount lustre-OST0000 complete [ 4601.786425] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8a2b03415000.ost_server_uuid 50 [ 4604.005805] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 4618.667767] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4618.674210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4618.685110] Lustre: Skipped 1 previous similar message [ 4618.698321] Lustre: Skipped 2 previous similar messages [ 4623.841752] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4623.854774] Lustre: Skipped 1 previous similar message [ 4623.961310] Lustre: server umount lustre-MDT0000 complete [ 4627.872817] LustreError: 178470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775851190 with bad export cookie 17580410751599681041 [ 4627.881285] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4627.897185] LustreError: 178470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4628.271077] Lustre: server umount lustre-MDT0001 complete [ 4636.943161] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 15:59:56 (1775851196) [ 4644.775601] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4645.176238] LustreError: 181973:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4645.276396] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4649.682402] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 4650.467702] LustreError: 181974:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4654.572327] LustreError: 181973:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4658.166712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4658.550970] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4663.309452] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 4669.544711] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4673.751655] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4681.255495] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4681.658164] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4689.819257] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 4689.910479] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4697.610940] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4706.393291] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4706.742271] Lustre: lustre-OST0001: new disk, initializing [ 4706.757655] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4706.917047] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4714.161887] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4714.178177] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4714.360447] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4715.056637] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 4724.114389] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4728.801895] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4728.818762] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4728.841986] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4733.538782] Lustre: server umount lustre-OST0000 complete [ 4734.956524] LustreError: 183570:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4734.976607] LustreError: 183570:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4740.142232] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4740.351567] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 4745.191422] LustreError: 184996:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4745.232303] LustreError: 184996:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 4745.569299] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4745.910540] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 4760.546433] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4760.546727] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4760.560790] LustreError: Skipped 1 previous similar message [ 4760.576989] Lustre: Skipped 2 previous similar messages [ 4760.583481] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4760.587685] Lustre: Skipped 4 previous similar messages [ 4765.670159] LustreError: 183569:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4765.691619] LustreError: 183569:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 4766.738813] Lustre: server umount lustre-OST0001 complete [ 4771.302694] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4771.309373] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4777.151875] Lustre: server umount lustre-MDT0000 complete [ 4780.913976] LustreError: 181953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775851343 with bad export cookie 17580410751599682112 [ 4780.915737] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4780.944337] LustreError: 181953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4781.399611] Lustre: server umount lustre-MDT0001 complete [ 4808.084225] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 16:02:47 (1775851367) [ 4816.532749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4817.046633] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4817.151927] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4817.449605] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4823.007908] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 4832.360961] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4832.457773] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4832.513421] Lustre: 187784:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4837.582219] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 4843.903219] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4848.756714] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4856.164588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4856.359712] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4863.605045] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 4863.705635] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 4870.488157] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4877.710988] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4877.773525] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4884.940299] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 4887.752508] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 4891.717886] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4897.984898] Lustre: lustre-OST0000: Not available for connect from 192.168.203.37@tcp (stopping) [ 4897.990856] Lustre: Skipped 5 previous similar messages [ 4898.291177] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4898.311581] Lustre: Skipped 2 previous similar messages [ 4901.514385] Lustre: server umount lustre-OST0000 complete [ 4903.062297] LustreError: 189406:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4918.754917] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4924.554855] Lustre: server umount lustre-OST0001 complete [ 4934.995858] Lustre: server umount lustre-MDT0000 complete [ 4939.533450] LustreError: 189427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775851502 with bad export cookie 17580410751599683519 [ 4939.543449] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4939.547083] LustreError: 189427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4940.203561] Lustre: server umount lustre-MDT0001 complete [ 4966.340816] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 16:05:26 (1775851526) [ 4980.653267] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_hostid [ 4991.367045] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 5028.522543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5028.857573] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5028.913318] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5029.035620] Lustre: lustre-MDT0000: new disk, initializing [ 5029.140936] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5029.147841] Lustre: Skipped 3 previous similar messages [ 5029.170422] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5033.428475] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 5045.749815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5045.937581] Lustre: 195395:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5046.023834] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5046.027470] Lustre: Skipped 1 previous similar message [ 5046.133643] Lustre: lustre-MDT0001: new disk, initializing [ 5046.326710] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5046.344319] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5052.589269] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 5061.320173] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5065.571550] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5073.684333] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5074.070824] Lustre: lustre-OST0000: new disk, initializing [ 5074.078091] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5080.941928] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 5082.745447] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5082.755281] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5082.904806] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5088.970211] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5093.248829] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5093.479549] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5099.186192] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5099.538413] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5103.587581] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5103.595156] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5103.596048] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5103.596053] Lustre: Skipped 10 previous similar messages [ 5103.623340] Lustre: Skipped 5 previous similar messages [ 5103.649408] LustreError: Skipped 2 previous similar messages [ 5116.386632] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5116.733845] Lustre: server umount lustre-OST0000 complete [ 5136.353120] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5136.524660] Lustre: server umount lustre-MDT0000 complete [ 5139.447307] LustreError: 195406:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5139.480459] LustreError: 195406:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 15 previous similar messages [ 5139.708025] LustreError: 195386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775851702 with bad export cookie 17580410751599685304 [ 5139.711036] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5139.718859] LustreError: 195386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5140.042347] Lustre: server umount lustre-MDT0001 complete [ 5147.900985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5153.576597] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 5162.332475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5162.639232] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5162.648270] Lustre: Skipped 3 previous similar messages [ 5167.505989] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 5173.752877] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5178.130401] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5185.435788] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5192.494422] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 5198.434147] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5203.669914] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5203.904248] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5209.112359] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5209.393322] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5213.667558] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5213.684443] LustreError: Skipped 1 previous similar message [ 5213.686682] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5213.697348] Lustre: Skipped 2 previous similar messages [ 5217.783262] Lustre: server umount lustre-OST0000 complete [ 5224.658982] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5229.787357] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5259.770386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5265.263406] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5265.274680] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5266.910953] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 5273.751711] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5277.959100] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 2 sec [ 5279.589964] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5279.603472] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5284.871924] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5284.877500] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5284.891731] LustreError: Skipped 1 previous similar message [ 5284.906796] Lustre: Skipped 15 previous similar messages [ 5289.629807] Lustre: server umount lustre-OST0000 complete [ 5300.790486] Lustre: server umount lustre-MDT0000 complete [ 5305.316759] LustreError: 202526:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5305.344609] LustreError: 202526:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 5305.689806] LustreError: 199188:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775851868 with bad export cookie 17580410751599686186 [ 5305.700964] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5305.712717] LustreError: 199188:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5306.053132] Lustre: server umount lustre-MDT0001 complete [ 5315.878761] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 16:11:15 (1775851875) [ 5323.399622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5329.026046] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 5339.045102] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5343.313495] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 5350.509680] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5356.137857] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5364.554367] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5372.110357] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 5379.201273] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5383.330870] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5383.530690] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5388.267815] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5388.624092] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5396.417735] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5396.616872] Lustre: lustre-OST0001: new disk, initializing [ 5396.620676] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5402.249738] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5402.278824] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5402.482933] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5402.855231] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 5412.319472] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5420.675709] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5420.934108] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5429.150048] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5429.676596] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5433.318943] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5433.349580] Lustre: Skipped 6 previous similar messages [ 5433.356551] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5438.254696] Lustre: server umount lustre-OST0001 complete [ 5446.321781] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5446.643289] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5452.167254] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5452.445914] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5483.181883] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5483.576822] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5483.581991] Lustre: Skipped 6 previous similar messages [ 5488.644235] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5488.666601] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5488.672369] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5490.391273] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 5496.515383] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5498.645264] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5515.232190] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5515.414186] Lustre: server umount lustre-OST0001 complete [ 5540.191550] Lustre: server umount lustre-OST0000 complete [ 5549.542362] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5549.552189] Lustre: Skipped 17 previous similar messages [ 5551.671377] Lustre: server umount lustre-MDT0000 complete [ 5556.352605] LustreError: 204641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775852118 with bad export cookie 17580410751599687152 [ 5556.364323] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5556.376582] LustreError: 204641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5562.713637] Lustre: server umount lustre-MDT0001 complete [ 5588.807297] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 16:15:48 (1775852148) [ 5596.297671] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5596.705944] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5596.734964] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5601.827654] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 5610.246556] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5610.335045] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5610.368178] Lustre: 212621:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5615.245047] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 5621.516763] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5625.911095] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5633.305499] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5633.421445] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5641.647780] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 5647.452635] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5659.018532] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5659.154768] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5666.078266] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 5672.232549] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5674.559325] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5679.716338] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5679.975038] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5685.049626] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5685.261745] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5689.295878] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8a2b20677000.ost_server_uuid 50 [ 5691.391069] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8a2b20677000.ost_server_uuid in IDLE state after 0 sec [ 5692.347467] Lustre: Permanently deactivating lustre-OST0001 [ 5692.385470] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 5696.960537] Lustre: Permanently reactivating lustre-OST0001 [ 5696.985718] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 5696.989663] Lustre: Skipped 2 previous similar messages [ 5703.158873] LustreError: lustre-OST0001-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 5703.170899] LustreError: Skipped 4 previous similar messages [ 5703.175604] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5703.188471] Lustre: Skipped 6 previous similar messages [ 5708.089408] Lustre: server umount lustre-OST0001 complete [ 5711.343316] LustreError: 214243:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5711.374157] LustreError: 214243:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 42 previous similar messages [ 5718.967730] Lustre: server umount lustre-OST0000 complete [ 5728.802423] Lustre: server umount lustre-MDT0000 complete [ 5732.450437] LustreError: 212614:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775852294 with bad export cookie 17580410751599688440 [ 5732.457286] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5733.015449] Lustre: server umount lustre-MDT0001 complete [ 5757.473854] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 16:18:37 (1775852317) [ 5765.579783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5765.909432] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5765.944304] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5765.958058] Lustre: Skipped 2 previous similar messages [ 5770.475547] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 5779.444504] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5779.659273] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5779.696664] Lustre: 218734:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5785.069304] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 5790.780841] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5794.852399] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5802.174627] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5802.448073] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5802.486292] Lustre: Permanently deactivating lustre-OST0000 [ 5802.499194] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 5807.942207] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 5812.777115] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5813.140945] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5820.130894] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5820.237175] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5823.011594] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 5825.968327] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 5831.400427] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5832.998705] Lustre: Permanently reactivating lustre-OST0000 [ 5842.473574] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 5842.479387] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5842.485940] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5842.490346] Lustre: Skipped 1 previous similar message [ 5842.510685] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 5842.523706] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5846.461774] Lustre: server umount lustre-OST0001 complete [ 5856.100062] Lustre: server umount lustre-OST0000 complete [ 5866.033587] Lustre: server umount lustre-MDT0000 complete [ 5870.112901] LustreError: 218729:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775852432 with bad export cookie 17580410751599690113 [ 5870.117079] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5870.129083] LustreError: 218729:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5870.583167] Lustre: server umount lustre-MDT0001 complete [ 5882.341425] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 5884.969647] Key type lgssc unregistered [ 5885.294560] LNet: 223121:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5885.304678] LNetError: 223121:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5885.326389] LNet: Removed LNI 192.168.203.137@tcp [ 5886.202726] Key type .llcrypt unregistered [ 5886.205080] Key type ._llcrypt unregistered [ 5908.426614] Key type ._llcrypt registered [ 5908.429921] Key type .llcrypt registered [ 5908.554369] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 5910.063671] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5910.089247] alg: No test for adler32 (adler32-zlib) [ 5911.256756] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [ 5911.526800] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 5913.248362] Key type lgssc registered [ 5914.549133] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5924.899457] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5924.913309] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5926.576483] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5931.607493] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 5937.126578] LustreError: 224570:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5937.140833] LustreError: 224570:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5940.596698] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5940.722864] Lustre: Found index 1 for lustre-MDT0001, updating log [ 5940.753732] Lustre: 224558:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5940.766685] Lustre: Permanently deactivating lustre-MDT0001 [ 5940.779592] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 5941.054958] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5943.057551] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 5943.073206] LustreError: 224602:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 5943.085325] LustreError: 224602:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 5943.102851] LustreError: 224602:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 16, retries 1, failed: rc = -108 [ 5945.060446] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 5951.222908] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5955.003449] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5962.467833] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5962.909205] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5966.006430] LustreError: 226188:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5969.705285] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 5971.435696] LustreError: 226188:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5972.535722] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 5975.669690] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5977.581862] LustreError: 226189:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5977.600408] LustreError: 226189:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5982.713366] LustreError: 226611:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5982.743367] LustreError: 226611:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5983.375128] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5983.571722] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5988.856750] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 5988.874952] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 5989.714734] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 5996.163329] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6000.334777] Lustre: Permanently reactivating lustre-MDT0001 [ 6000.371043] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 6000.385353] Lustre: Skipped 2 previous similar messages [ 6005.438168] Lustre: lustre-MDT0001: Client 1e0fb545-b61e-4538-bd08-bf9c1b872dbf (at 192.168.203.37@tcp) reconnecting [ 6010.142333] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6012.102452] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6017.505446] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6017.703541] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6021.859548] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6022.045504] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6024.568928] Lustre: Permanently deactivating lustre-MDT0001 [ 6024.589870] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6024.595833] Lustre: Skipped 2 previous similar messages [ 6033.515032] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6043.617349] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6043.623297] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6043.633899] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6045.155641] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6045.158217] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6045.167963] Lustre: Skipped 2 previous similar messages [ 6045.181361] Lustre: Skipped 3 previous similar messages [ 6048.730840] Lustre: server umount lustre-MDT0000 complete [ 6050.279783] LustreError: 224566:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6050.297802] LustreError: 224566:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6051.925319] LustreError: 224552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775852614 with bad export cookie 13782015768978148477 [ 6051.927941] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6051.935568] LustreError: 224552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6052.308970] Lustre: server umount lustre-MDT0001 complete [ 6058.690830] Lustre: server umount lustre-OST0000 complete [ 6071.077310] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 16:23:51 (1775852631) [ 6082.481151] Lustre: server umount lustre-OST0001 complete [ 6090.969939] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_hostid [ 6105.919840] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 6148.185882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6148.516914] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6148.525987] Lustre: Skipped 2 previous similar messages [ 6148.590039] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6148.701576] Lustre: lustre-MDT0000: new disk, initializing [ 6148.814436] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6148.853491] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6154.616383] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 6167.058300] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6167.197900] Lustre: 232297:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6167.288351] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6167.296870] Lustre: Skipped 1 previous similar message [ 6167.436455] Lustre: lustre-MDT0001: new disk, initializing [ 6167.637935] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6167.677972] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6167.703593] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6173.094694] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 6183.311888] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6187.941866] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6196.892617] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6197.217641] Lustre: lustre-OST0000: new disk, initializing [ 6197.223515] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6197.366438] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6203.672379] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 6206.553297] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6206.566568] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6206.714628] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6211.583201] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6214.456612] LustreError: 232305:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6223.548645] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6223.731803] Lustre: lustre-OST0001: new disk, initializing [ 6223.740845] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6224.464888] LustreError: 232305:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6224.503034] LustreError: 232304:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6230.148157] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6230.164324] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6230.205028] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 6230.298267] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6234.561077] LustreError: 232304:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6238.773535] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6240.843971] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6245.866373] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6245.879268] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6245.901163] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6245.907141] LustreError: Skipped 1 previous similar message [ 6245.934771] Lustre: Skipped 1 previous similar message [ 6249.363268] Lustre: server umount lustre-OST0001 complete [ 6250.989237] LustreError: 234496:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6251.012378] LustreError: 234496:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6268.390050] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6268.392136] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6268.400906] LustreError: Skipped 1 previous similar message [ 6268.406273] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6268.423205] Lustre: Skipped 2 previous similar messages [ 6268.465262] Lustre: Skipped 1 previous similar message [ 6276.579348] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6276.590738] Lustre: Skipped 3 previous similar messages [ 6282.208098] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6282.563944] Lustre: server umount lustre-OST0000 complete [ 6286.818723] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6286.844139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6286.847224] Lustre: Skipped 2 previous similar messages [ 6301.153635] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6301.539675] Lustre: server umount lustre-MDT0000 complete [ 6302.180035] LustreError: 232308:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6302.191017] LustreError: 232308:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 13 previous similar messages [ 6305.950409] LustreError: 232290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775852868 with bad export cookie 13782015768978151067 [ 6305.977337] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6305.978991] LustreError: 232290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6312.569197] Lustre: server umount lustre-MDT0001 complete [ 6325.561215] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 6328.953822] Key type lgssc unregistered [ 6329.333885] LNet: 236949:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6329.349774] LNetError: 236949:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6329.375458] LNet: Removed LNI 192.168.203.137@tcp [ 6330.323207] Key type .llcrypt unregistered [ 6330.325223] Key type ._llcrypt unregistered [ 6375.323910] Key type ._llcrypt registered [ 6375.325766] Key type .llcrypt registered [ 6375.431681] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 6376.622515] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6376.787520] alg: No test for adler32 (adler32-zlib) [ 6377.998854] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [ 6378.302321] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 6380.016170] Key type lgssc registered [ 6381.129214] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6388.573549] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6388.611250] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6390.032568] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6390.046887] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6390.275655] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6395.728093] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 6406.794993] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6406.887284] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6406.938975] Lustre: 239142:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6407.353018] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6413.346442] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 6420.788881] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6425.618588] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6433.031641] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6433.227976] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6433.514764] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6439.078925] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 6440.389613] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 6447.415737] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6505.449074] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6505.458614] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6505.485212] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6505.962826] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6505.970683] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6505.988465] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6509.613566] Lustre: server umount lustre-OST0000 complete [ 6514.985167] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 6529.509028] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6530.166656] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6537.082559] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 6543.468368] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6545.532869] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6545.553591] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6545.553636] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6545.576719] Lustre: Skipped 1 previous similar message [ 6586.850619] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6586.859154] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6586.869651] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6586.871891] Lustre: Skipped 1 previous similar message [ 6586.882933] Lustre: Skipped 1 previous similar message [ 6590.406393] Lustre: server umount lustre-OST0000 complete [ 6596.577054] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6596.586963] LustreError: Skipped 1 previous similar message [ 6596.591328] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6596.614610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6600.872758] Lustre: server umount lustre-MDT0000 complete [ 6604.692282] LustreError: 239133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775853167 with bad export cookie 13319818786439716321 [ 6604.693501] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6604.701086] LustreError: 239133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6605.184724] Lustre: server umount lustre-MDT0001 complete [ 6616.751854] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 6619.416594] Key type lgssc unregistered [ 6619.699364] LNet: 244105:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6619.709122] LNetError: 244105:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6619.722423] LNet: Removed LNI 192.168.203.137@tcp [ 6620.488615] Key type .llcrypt unregistered [ 6620.492086] Key type ._llcrypt unregistered [ 6651.386646] Key type ._llcrypt registered [ 6651.391125] Key type .llcrypt registered [ 6651.538653] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 6653.634983] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6653.717406] alg: No test for adler32 (adler32-zlib) [ 6654.849145] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [ 6655.157879] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 6656.848166] Key type lgssc registered [ 6658.735633] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6669.468732] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6669.497809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6671.375569] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6676.517567] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 6680.549845] LustreError: 245508:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6680.584266] LustreError: 245508:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6685.665943] LustreError: 245507:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6687.541818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6687.973664] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6694.999788] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 6704.076913] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6709.193810] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6718.100744] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6718.654748] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6724.838462] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 6730.808086] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6741.184309] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 6769.128219] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6769.139395] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6769.167198] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6772.196381] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6772.197253] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6772.199458] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6772.226610] Lustre: Skipped 1 previous similar message [ 6773.506432] Lustre: server umount lustre-OST0000 complete [ 6777.316408] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6777.330308] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6782.433824] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6782.448056] Lustre: Skipped 2 previous similar messages [ 6783.549131] Lustre: server umount lustre-MDT0000 complete [ 6787.554980] LustreError: 245507:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6787.583040] LustreError: 245507:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6787.806190] LustreError: 245489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775853350 with bad export cookie 152955503127416353 [ 6787.813322] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6787.818168] LustreError: 245489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6788.311892] Lustre: server umount lustre-MDT0001 complete [ 6799.553543] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 6802.706908] Key type lgssc unregistered [ 6802.999235] LNet: 249486:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6803.011963] LNetError: 249486:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6803.033378] LNet: Removed LNI 192.168.203.137@tcp [ 6803.865112] Key type .llcrypt unregistered [ 6803.867105] Key type ._llcrypt unregistered [ 6821.168853] Key type ._llcrypt registered [ 6821.172316] Key type .llcrypt registered [ 6821.271605] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 6822.400452] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6822.437698] alg: No test for adler32 (adler32-zlib) [ 6823.495369] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [ 6823.717385] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 6825.384237] Key type lgssc registered [ 6826.496965] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6833.996813] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6834.022272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6835.378127] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6839.980801] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 6844.914270] LustreError: 250691:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6844.943851] LustreError: 250691:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6849.935730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6850.022536] LustreError: 250690:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6850.408347] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6854.941594] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 6861.760121] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6865.788995] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6873.049100] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6873.481718] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6879.560339] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 6879.673193] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6886.558367] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6905.315470] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6905.333364] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6905.334116] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6905.354319] Lustre: Skipped 1 previous similar message [ 6905.374940] Lustre: Skipped 1 previous similar message [ 6906.747384] Lustre: server umount lustre-OST0000 complete [ 6911.975450] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6911.987537] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6912.004470] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6915.556573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6915.559914] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6917.485459] Lustre: server umount lustre-MDT0000 complete [ 6920.678660] LustreError: 250690:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6920.696174] LustreError: 250690:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6921.682647] LustreError: 250672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775853484 with bad export cookie 13660267997170820267 [ 6921.696220] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6921.699167] LustreError: 250672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6922.299590] Lustre: server umount lustre-MDT0001 complete [ 6934.630366] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 6937.330687] Key type lgssc unregistered [ 6937.596281] LNet: 254015:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6937.605286] LNetError: 254015:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6937.622803] LNet: Removed LNI 192.168.203.137@tcp [ 6938.622583] Key type .llcrypt unregistered [ 6938.624349] Key type ._llcrypt unregistered [ 6962.751414] Key type ._llcrypt registered [ 6962.755235] Key type .llcrypt registered [ 6962.873875] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 6964.240111] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6964.264678] alg: No test for adler32 (adler32-zlib) [ 6965.354540] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [ 6965.588485] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 6967.268176] Key type lgssc registered [ 6968.537450] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6977.233355] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6977.250150] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6978.949961] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6983.834647] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 6989.281334] LustreError: 255414:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6989.294453] LustreError: 255414:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6990.307102] LustreError: 255415:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6994.455382] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6994.947949] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7000.915372] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 7007.309924] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7011.515864] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7018.258292] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7018.516580] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7024.469903] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 7024.695972] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7030.977420] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7055.336026] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7055.339593] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7055.342261] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7055.348151] LustreError: Skipped 1 previous similar message [ 7055.375231] Lustre: Skipped 1 previous similar message [ 7060.455016] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7060.460423] Lustre: Skipped 2 previous similar messages [ 7061.084526] Lustre: server umount lustre-OST0000 complete [ 7065.569668] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7065.572769] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7065.578238] Lustre: Skipped 1 previous similar message [ 7070.689882] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7070.700066] Lustre: Skipped 2 previous similar messages [ 7071.649172] Lustre: server umount lustre-MDT0000 complete [ 7075.796937] LustreError: 255394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775853638 with bad export cookie 7204921509699020472 [ 7075.799899] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7075.821622] LustreError: 255394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7076.216529] Lustre: server umount lustre-MDT0001 complete [ 7090.022202] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 7093.181902] Key type lgssc unregistered [ 7093.454853] LNet: 259369:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7093.468620] LNetError: 259369:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7093.493880] LNet: Removed LNI 192.168.203.137@tcp [ 7094.408803] Key type .llcrypt unregistered [ 7094.410714] Key type ._llcrypt unregistered [ 7113.735408] Key type ._llcrypt registered [ 7113.737572] Key type .llcrypt registered [ 7113.858223] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 7115.654620] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7115.762228] alg: No test for adler32 (adler32-zlib) [ 7116.884802] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [ 7117.165604] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 7118.816231] Key type lgssc registered [ 7119.967580] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7128.964821] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7128.994336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7130.849990] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7136.267589] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 7141.350080] LustreError: 260605:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7141.365582] LustreError: 260605:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7146.166172] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7146.470874] LustreError: 260607:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7146.935785] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7152.206691] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 7158.108574] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7161.601641] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7169.003321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7169.412072] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7175.316472] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 7175.533451] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 7180.959932] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7193.059766] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7193.068944] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7193.078215] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7196.133277] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7196.143351] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7196.144824] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7196.170791] Lustre: Skipped 1 previous similar message [ 7198.702568] Lustre: server umount lustre-OST0000 complete [ 7203.305417] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7203.318102] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7203.329643] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7206.388842] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7206.394572] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7206.427037] Lustre: Skipped 1 previous similar message [ 7211.494767] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7211.503177] Lustre: Skipped 1 previous similar message [ 7217.120147] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7217.409262] Lustre: server umount lustre-MDT0000 complete [ 7220.711385] LustreError: 260595:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7220.721650] LustreError: 260595:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7221.382788] LustreError: 260572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775853783 with bad export cookie 10736798608101598683 [ 7221.396654] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7221.401728] LustreError: 260572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7222.156788] Lustre: server umount lustre-MDT0001 complete [ 7234.922274] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 7237.597710] Key type lgssc unregistered [ 7238.062164] LNet: 263955:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7238.071226] LNetError: 263955:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7238.131639] LNet: Removed LNI 192.168.203.137@tcp [ 7239.283085] Key type .llcrypt unregistered [ 7239.286107] Key type ._llcrypt unregistered [ 7254.586880] Key type ._llcrypt registered [ 7254.592671] Key type .llcrypt registered [ 7254.769861] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 7272.717192] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_hostid [ 7288.559511] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 7289.533697] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7289.579724] alg: No test for adler32 (adler32-zlib) [ 7290.666145] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [ 7290.864117] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 7292.576182] Key type lgssc registered [ 7293.764547] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7329.686524] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7329.715720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7330.952763] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7331.004157] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7331.091141] Lustre: lustre-MDT0000: new disk, initializing [ 7331.163349] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7331.178919] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7334.525780] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 7344.169029] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7344.298764] Lustre: 267458:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7344.409449] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7344.425936] Lustre: Skipped 1 previous similar message [ 7344.559946] Lustre: lustre-MDT0001: new disk, initializing [ 7344.687939] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7344.741297] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7344.749451] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7349.004381] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 7356.317296] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7359.998437] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7366.842665] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7367.065939] Lustre: lustre-OST0000: new disk, initializing [ 7367.071054] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7367.157335] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7372.283193] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 7375.927320] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7375.941943] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7376.030799] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7378.449429] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7381.795489] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7381.996130] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7385.762645] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7385.952030] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7391.719677] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7391.729034] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7391.746750] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7391.762182] Lustre: Skipped 1 previous similar message [ 7393.957832] Lustre: server umount lustre-OST0000 complete [ 7400.929997] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7400.940892] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7400.950790] Lustre: Skipped 1 previous similar message [ 7400.955774] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7401.959242] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7401.961252] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7403.937370] Lustre: server umount lustre-MDT0000 complete [ 7407.406187] LustreError: 267450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775853969 with bad export cookie 3095953753080176465 [ 7407.418609] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7407.424111] LustreError: 267450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7407.766912] Lustre: server umount lustre-MDT0001 complete [ 7415.180662] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 16:46:15 (1775853975) [ 7420.530690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7421.012413] LustreError: 271466:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7421.036378] LustreError: 271466:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7421.098964] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7424.805721] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 7426.531789] LustreError: 271467:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7430.630594] LustreError: 271466:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7434.025713] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7434.539994] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7438.573097] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 7443.671879] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7448.141098] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7454.074877] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7454.351798] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7458.916670] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 7464.921864] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7475.170513] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7475.181717] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7475.201038] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7475.213499] Lustre: Skipped 1 previous similar message [ 7480.849390] Lustre: server umount lustre-OST0000 complete [ 7485.921237] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7485.937066] LustreError: Skipped 1 previous similar message [ 7485.944119] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7485.960747] Lustre: Skipped 1 previous similar message [ 7485.976755] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7485.984819] Lustre: Skipped 2 previous similar messages [ 7496.683769] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7496.688905] Lustre: Skipped 5 previous similar messages [ 7499.232129] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7499.384351] Lustre: server umount lustre-MDT0000 complete [ 7500.771908] LustreError: 271466:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7500.799216] LustreError: 271466:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7503.215189] LustreError: 271448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775854065 with bad export cookie 3095953753080177347 [ 7503.242403] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7503.833574] Lustre: server umount lustre-MDT0001 complete [ 7514.377530] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 7517.033792] Key type lgssc unregistered [ 7517.455684] LNet: 274648:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7517.482747] LNetError: 274648:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7517.502339] LNet: Removed LNI 192.168.203.137@tcp [ 7518.398150] Key type .llcrypt unregistered [ 7518.400142] Key type ._llcrypt unregistered [ 7547.033796] Key type ._llcrypt registered [ 7547.036439] Key type .llcrypt registered [ 7547.136496] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_hostid [ 7562.088979] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 7563.213750] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7563.284360] alg: No test for adler32 (adler32-zlib) [ 7564.430856] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [ 7564.773968] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 7566.544180] Key type lgssc registered [ 7568.162799] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7614.891642] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7614.943340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7616.208518] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7616.237883] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7616.324200] Lustre: lustre-MDT0000: new disk, initializing [ 7616.410483] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7616.446494] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7620.836794] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 7630.583206] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7630.682109] Lustre: 278443:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7630.753359] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7630.757760] Lustre: Skipped 1 previous similar message [ 7630.820638] Lustre: lustre-MDT0001: new disk, initializing [ 7630.894867] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7630.932405] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7630.952412] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7635.060683] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 7641.777542] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7645.413460] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7652.286166] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7652.660480] Lustre: lustre-OST2710: new disk, initializing [ 7652.664431] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 7652.757066] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 7658.666305] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 7661.659405] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 7661.673677] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 7661.844202] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 7666.308198] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 7672.430201] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7672.559073] Lustre: lustre-OST03e8: new disk, initializing [ 7672.562249] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 7672.626234] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 7678.769845] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 7680.063913] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 7680.072775] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 7680.155302] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 7684.193729] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 7690.695917] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 7690.990351] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7695.604577] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 7695.854030] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7700.361862] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 7700.607269] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7704.986303] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 7705.296976] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7711.214920] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7711.226238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7711.227474] Lustre: Skipped 2 previous similar messages [ 7711.241775] Lustre: Skipped 3 previous similar messages [ 7712.226469] Lustre: server umount lustre-MDT0000 complete [ 7715.631585] LustreError: 278436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775854278 with bad export cookie 11166598881815820887 [ 7715.642823] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7715.649957] LustreError: 278436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7716.017828] Lustre: server umount lustre-MDT0001 complete [ 7728.672821] Lustre: server umount lustre-OST2710 complete [ 7731.616109] Lustre: 276240:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775854278/real 1775854278] req@00000000abf9bb0f x1862118017135616/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1775854294 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7731.636395] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7731.648902] Lustre: Skipped 1 previous similar message [ 7732.380577] Lustre: server umount lustre-OST03e8 complete [ 7750.856656] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_hostid [ 7760.551758] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 7790.362657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7790.543214] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7790.580461] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7790.658824] Lustre: lustre-MDT0000: new disk, initializing [ 7790.725517] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7790.742579] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7795.156640] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 7806.935821] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7807.061656] Lustre: 286165:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7807.148763] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7807.155720] Lustre: Skipped 1 previous similar message [ 7807.292285] Lustre: lustre-MDT0001: new disk, initializing [ 7807.355339] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7807.382316] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7807.393952] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7811.823449] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 7818.975952] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7822.387518] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7829.108053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7829.419284] Lustre: lustre-OST0000: new disk, initializing [ 7829.426976] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7829.523565] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7835.509886] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 7837.292811] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7837.306438] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7837.392511] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7843.474799] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7848.355599] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7848.534688] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7853.125697] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7853.368802] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7858.150162] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7858.157979] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7858.174348] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7858.193642] Lustre: Skipped 1 previous similar message [ 7861.530859] Lustre: server umount lustre-OST0000 complete [ 7868.395070] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7868.410594] Lustre: Skipped 2 previous similar messages [ 7868.416310] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7868.426925] Lustre: Skipped 1 previous similar message [ 7873.663508] Lustre: server umount lustre-MDT0000 complete [ 7877.500864] LustreError: 286156:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775854440 with bad export cookie 11166598881815822567 [ 7877.504120] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7877.514590] LustreError: 286156:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7877.845867] Lustre: server umount lustre-MDT0001 complete [ 7887.480441] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 16:54:07 (1775854447) [ 7889.341985] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 7891.481930] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 16:54:11 (1775854451) [ 7895.366627] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 7926.542568] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7926.795592] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7926.825655] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7926.985591] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7931.755311] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 7940.648354] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7940.791988] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7940.838911] Lustre: 291601:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7944.787293] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 7950.212760] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7954.486772] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7974.252669] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7974.451970] Lustre: Denying initial registration attempt from nid 192.168.203.137@tcp, specified as failover [ 7974.461862] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 7974.480315] LustreError: 293211:0:(tgt_mount.c:2545:server_fill_super()) Unable to start targets: -99 [ 7974.486812] LustreError: 293211:0:(tgt_mount.c:2063:server_put_super()) no obd lustre-OST0000 [ 7974.495655] LustreError: 293211:0:(tgt_mount.c:114:server_deregister_mount()) lustre-OST0000 not registered [ 7974.580888] Lustre: server umount lustre-OST0000 complete [ 7974.584289] LustreError: 293211:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 7984.098924] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7984.137723] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7984.147762] Lustre: Skipped 1 previous similar message [ 7988.752422] Lustre: server umount lustre-MDT0000 complete [ 7989.217095] LustreError: 292443:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7989.246568] LustreError: 292443:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 7993.029725] LustreError: 291592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775854555 with bad export cookie 11166598881815823442 [ 7993.043365] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7993.054521] LustreError: 291592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7993.365115] Lustre: server umount lustre-MDT0001 complete [ 8004.390830] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 8006.872487] Key type lgssc unregistered [ 8007.183701] LNet: 294316:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8007.195556] LNetError: 294316:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8007.206672] LNet: Removed LNI 192.168.203.137@tcp [ 8007.980187] Key type .llcrypt unregistered [ 8007.982052] Key type ._llcrypt unregistered [ 8021.186857] Key type ._llcrypt registered [ 8021.190125] Key type .llcrypt registered [ 8021.284967] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 8022.519681] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8022.543896] alg: No test for adler32 (adler32-zlib) [ 8023.661758] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [ 8023.850673] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 8025.560164] Key type lgssc registered [ 8026.738033] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8082.996369] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 8090.568561] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8090.592890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8091.896031] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8091.927909] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8092.104920] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8095.841959] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 8104.667310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8104.764136] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8104.820607] Lustre: 297058:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8105.073702] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8109.130989] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 8115.258364] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8119.179621] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8139.439860] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8139.679165] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8139.918084] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8146.476251] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 8151.904348] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8155.118406] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8155.125117] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8155.126146] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8155.160945] Lustre: Skipped 1 previous similar message [ 8160.046404] Lustre: server umount lustre-OST0000 complete [ 8165.347107] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8165.361959] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8165.369513] Lustre: Skipped 2 previous similar messages [ 8170.548337] Lustre: server umount lustre-MDT0000 complete [ 8174.467440] LustreError: 297048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775854736 with bad export cookie 7314719577688058088 [ 8174.472516] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8174.480598] LustreError: 297048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8174.843685] Lustre: server umount lustre-MDT0001 complete [ 8186.476271] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 8189.324300] Key type lgssc unregistered [ 8189.644641] LNet: 300273:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8189.652944] LNetError: 300273:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8189.675162] LNet: Removed LNI 192.168.203.137@tcp [ 8190.473695] Key type .llcrypt unregistered [ 8190.475559] Key type ._llcrypt unregistered [ 8216.018291] Key type ._llcrypt registered [ 8216.021703] Key type .llcrypt registered [ 8216.118840] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 8217.270375] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8217.295655] alg: No test for adler32 (adler32-zlib) [ 8218.356637] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [ 8218.575862] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 8220.232740] Key type lgssc registered [ 8221.231404] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8229.037869] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8229.065371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8230.617249] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8235.221343] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 8241.130065] LustreError: 301677:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8241.153415] LustreError: 301677:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8243.344228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8243.856979] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8248.295279] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 8255.035383] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8259.892203] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8269.103682] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8269.525343] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8275.946876] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 8282.293368] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8299.493303] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8299.500804] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8299.515333] Lustre: Skipped 2 previous similar messages [ 8299.528312] Lustre: Skipped 2 previous similar messages [ 8300.726407] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.37@tcp (stopping) [ 8303.523948] Lustre: server umount lustre-MDT0000 complete [ 8304.610071] LustreError: 301678:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8304.637293] LustreError: 301678:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8305.817939] LustreError: 302671:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8307.030985] LustreError: 301658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775854869 with bad export cookie 3060848420165052600 [ 8307.043551] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8307.049941] LustreError: 301658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8309.731170] LustreError: 301674:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8309.732683] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8309.781898] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8312.738044] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8312.746645] Lustre: Skipped 1 previous similar message [ 8313.491961] Lustre: server umount lustre-MDT0001 complete [ 8320.253831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8329.264363] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8332.257025] LustreError: 300986:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000099dc3d20 x1862118702882304/t0(0) o250->MGC192.168.203.137@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8332.542260] LustreError: 303282:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8332.557120] LustreError: 303282:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8332.650546] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8332.719198] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 8337.272225] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 8341.863740] LustreError: 304968:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8341.877749] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8341.888567] LustreError: 304968:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8346.482362] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8346.893035] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8351.703537] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 8352.237180] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8360.241947] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8362.820559] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8367.912801] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8370.211116] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8377.826413] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8377.827897] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8377.836460] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8377.864161] Lustre: Skipped 1 previous similar message [ 8377.880156] Lustre: Skipped 1 previous similar message [ 8382.852046] Lustre: server umount lustre-OST0000 complete [ 8388.070745] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8388.074193] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8388.084878] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8388.100845] Lustre: Skipped 1 previous similar message [ 8388.119494] Lustre: Skipped 1 previous similar message [ 8401.376131] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8401.688161] Lustre: server umount lustre-MDT0000 complete [ 8403.431331] LustreError: 304971:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8403.462709] LustreError: 304971:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8405.981099] LustreError: 301657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775854968 with bad export cookie 3060848420165060146 [ 8405.997831] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8405.999312] LustreError: 301657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 8406.443712] Lustre: server umount lustre-MDT0001 complete [ 8420.340640] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 8423.767392] Key type lgssc unregistered [ 8424.166520] LNet: 307406:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8424.175353] LNetError: 307406:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8424.194743] LNet: Removed LNI 192.168.203.137@tcp [ 8425.134131] Key type .llcrypt unregistered [ 8425.139058] Key type ._llcrypt unregistered [ 8452.377340] Key type ._llcrypt registered [ 8452.381691] Key type .llcrypt registered [ 8452.520268] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 8453.913871] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8454.116260] alg: No test for adler32 (adler32-zlib) [ 8455.231704] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [ 8455.525225] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 8457.192170] Key type lgssc registered [ 8458.448412] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8467.456403] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8467.487984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8469.227351] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8474.606188] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 8478.698262] LustreError: 308808:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8478.717865] LustreError: 308808:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8483.814307] LustreError: 308807:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8485.496174] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8486.010237] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8491.958366] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 8499.434598] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8504.658996] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8510.946161] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8510.965949] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8510.986379] Lustre: Skipped 1 previous similar message [ 8511.969480] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8511.984853] Lustre: Skipped 1 previous similar message [ 8514.347396] Lustre: server umount lustre-MDT0000 complete [ 8517.091913] LustreError: 308807:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8517.110995] LustreError: 308807:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8518.501850] LustreError: 308788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775855081 with bad export cookie 10360766348647944663 [ 8518.527971] LustreError: 308788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8518.528611] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8518.808367] Lustre: server umount lustre-MDT0001 complete [ 8534.940366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8535.216633] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8535.250579] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8535.468213] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8540.160324] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 8549.780689] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8550.008775] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8550.055410] Lustre: 311167:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8550.389871] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8554.620725] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 8561.219633] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8565.834501] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8576.132714] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8576.351182] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 8576.362187] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 8576.369725] LustreError: 312825:0:(tgt_mount.c:2545:server_fill_super()) Unable to start targets: -2 [ 8576.381629] LustreError: 312825:0:(tgt_mount.c:2063:server_put_super()) no obd lustre-OST0000 [ 8576.388899] LustreError: 312825:0:(tgt_mount.c:114:server_deregister_mount()) lustre-OST0000 not registered [ 8576.501114] Lustre: server umount lustre-OST0000 complete [ 8576.503132] LustreError: 312825:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 8587.440936] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8587.718465] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8587.920973] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8594.671063] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 8598.170896] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 8601.445574] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8608.810040] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8608.884322] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 8608.980286] Lustre: lustre-OST0001: new disk, initializing [ 8608.987840] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8609.099810] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8615.623824] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 8616.607242] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8616.619739] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8616.805317] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8622.592692] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 8627.178963] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8627.191842] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8627.195539] Lustre: Skipped 2 previous similar messages [ 8627.210912] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8632.020064] Lustre: server umount lustre-OST0001 complete [ 8632.294573] LustreError: 313784:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8632.320676] LustreError: 313784:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8637.413195] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8637.423706] LustreError: 313294:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8637.461284] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8637.461292] Lustre: Skipped 1 previous similar message [ 8637.501853] LustreError: 313294:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8642.532781] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8642.552058] Lustre: Skipped 2 previous similar messages [ 8643.490218] Lustre: server umount lustre-OST0000 complete [ 8652.778136] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8652.778195] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8652.803121] Lustre: Skipped 1 previous similar message [ 8652.803926] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8652.841974] Lustre: Skipped 1 previous similar message [ 8656.316379] Lustre: server umount lustre-MDT0000 complete [ 8657.892778] LustreError: 311179:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8657.917212] LustreError: 311179:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8660.822447] LustreError: 311161:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775855223 with bad export cookie 10360766348647945216 [ 8660.829447] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8660.848752] LustreError: 311161:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8661.224774] Lustre: server umount lustre-MDT0001 complete [ 8673.886723] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 8677.229828] Key type lgssc unregistered [ 8677.576704] LNet: 316084:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8677.582138] LNetError: 316084:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8677.605833] LNet: Removed LNI 192.168.203.137@tcp [ 8678.678986] Key type .llcrypt unregistered [ 8678.681341] Key type ._llcrypt unregistered [ 8753.847986] Key type ._llcrypt registered [ 8753.853710] Key type .llcrypt registered [ 8754.008598] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_hostid [ 8768.122192] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 8768.920083] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8768.992899] alg: No test for adler32 (adler32-zlib) [ 8770.151058] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [ 8770.371971] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 8772.072349] Key type lgssc registered [ 8773.505074] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8815.320055] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8815.358314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8816.686922] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8816.719314] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8816.792929] Lustre: lustre-MDT0000: new disk, initializing [ 8816.893995] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8816.912365] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8821.302622] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 8832.419827] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8832.529746] Lustre: 320988:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8832.573513] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8832.577176] Lustre: Skipped 1 previous similar message [ 8832.633334] Lustre: lustre-MDT0001: new disk, initializing [ 8832.707680] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8832.761127] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8832.768819] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8837.062391] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 8844.594982] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8848.834872] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8855.464848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8855.901662] Lustre: lustre-OST0000: new disk, initializing [ 8855.908242] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8855.995821] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8862.499739] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 8864.318501] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8864.330456] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8864.529854] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8871.050896] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8876.601291] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8876.851355] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8881.534275] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8881.739684] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8885.218684] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8885.234229] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8885.254728] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8885.260355] Lustre: Skipped 1 previous similar message [ 8889.722637] Lustre: server umount lustre-OST0000 complete [ 8894.433522] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8894.438825] LustreError: Skipped 1 previous similar message [ 8894.450440] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8894.462560] Lustre: Skipped 1 previous similar message [ 8894.486766] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8895.461910] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8900.931324] Lustre: server umount lustre-MDT0000 complete [ 8905.462850] LustreError: 320981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775855467 with bad export cookie 9328985218619955736 [ 8905.472568] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8905.481305] LustreError: 320981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8905.816670] Lustre: server umount lustre-MDT0001 complete [ 8915.384401] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 17:11:15 (1775855475) [ 8924.248704] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 17:11:24 (1775855484) [ 8934.545175] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8935.143894] LustreError: 325338:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8935.172438] LustreError: 325338:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 8935.219931] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8939.821751] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 8940.512952] LustreError: 325339:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8944.617875] LustreError: 325338:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8948.918206] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8949.265394] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8953.555757] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 8960.418190] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8964.853621] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8973.114515] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8973.540103] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8980.807456] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 8987.376244] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8995.983928] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 8998.017758] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 8999.981199] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9001.833664] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 9003.680791] Lustre: Failing over lustre-MDT0000 [ 9003.914622] Lustre: server umount lustre-MDT0000 complete [ 9005.544861] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9005.557315] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9005.575294] LustreError: 325339:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9010.833929] LustreError: 325333:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9010.855497] LustreError: 325333:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 9021.074047] LustreError: 325333:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9021.088159] LustreError: 325333:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 9022.634735] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9022.822867] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9023.158033] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9023.242322] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9027.219514] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9028.327429] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 9028.607485] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9028.659805] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 9028.746661] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9036.692871] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9038.413207] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9042.303457] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 9044.339749] Lustre: Failing over lustre-MDT0000 [ 9044.739862] Lustre: server umount lustre-MDT0000 complete [ 9047.225199] LustreError: 326451:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9047.246000] LustreError: 326451:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 9049.059425] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9049.088223] Lustre: Skipped 3 previous similar messages [ 9049.103076] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9064.043494] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9064.201742] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9064.529310] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9064.621955] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9067.666168] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9068.936951] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 9069.558149] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9069.567711] Lustre: Skipped 2 previous similar messages [ 9069.608129] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 9069.646861] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 9075.245812] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9076.460196] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9084.900642] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9084.901476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9084.917030] Lustre: Skipped 3 previous similar messages [ 9084.929455] Lustre: Skipped 4 previous similar messages [ 9088.168717] Lustre: server umount lustre-MDT0000 complete [ 9090.024502] LustreError: 325334:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9090.038533] LustreError: 325334:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 15 previous similar messages [ 9091.482356] LustreError: 325318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775855654 with bad export cookie 9328985218619958445 [ 9091.504701] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9091.772946] Lustre: server umount lustre-MDT0001 complete [ 9106.532322] Lustre: server umount lustre-OST0000 complete [ 9126.307833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9133.582898] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 9144.517440] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9144.782718] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9144.797679] Lustre: Skipped 1 previous similar message [ 9149.898504] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 9156.348144] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9160.025092] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9166.675270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9172.152922] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 9173.101850] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 9179.289874] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9185.775261] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9185.784755] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9185.801658] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9186.612818] Lustre: server umount lustre-OST0000 complete [ 9190.885085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9200.185993] Lustre: server umount lustre-MDT0000 complete [ 9203.171314] LustreError: 331122:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9203.201048] LustreError: 331122:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 9204.678395] LustreError: 332745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775855767 with bad export cookie 9328985218619959166 [ 9204.685609] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9204.690770] LustreError: 332745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 9205.159819] Lustre: server umount lustre-MDT0001 complete [ 9217.095589] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 9219.814679] Key type lgssc unregistered [ 9220.137654] LNet: 334307:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9220.153825] LNetError: 334307:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9220.190604] LNet: Removed LNI 192.168.203.137@tcp [ 9221.272865] Key type .llcrypt unregistered [ 9221.275493] Key type ._llcrypt unregistered [ 9246.163201] Key type ._llcrypt registered [ 9246.167943] Key type .llcrypt registered [ 9246.364137] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 9248.183971] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9248.206698] alg: No test for adler32 (adler32-zlib) [ 9249.363381] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [ 9249.603523] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 9251.337663] Key type lgssc registered [ 9252.699496] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9261.354255] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9261.374217] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9262.935901] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9267.620306] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 9273.317737] LustreError: 335812:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9273.350813] LustreError: 335812:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9276.169367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9276.633385] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9281.075370] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 9286.715797] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9290.061943] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9296.221377] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9296.533938] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9301.651396] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 9307.140173] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9307.819198] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 9322.979041] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9322.988993] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9323.001501] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9323.492178] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9323.505278] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9323.506044] Lustre: Skipped 1 previous similar message [ 9323.524538] Lustre: Skipped 2 previous similar messages [ 9325.937862] Lustre: server umount lustre-MDT0000 complete [ 9328.613304] LustreError: 335807:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9328.635042] LustreError: 335807:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9329.572666] LustreError: 335793:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775855892 with bad export cookie 3855072152987927506 [ 9329.586800] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9329.587564] LustreError: 335793:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9329.862560] Lustre: server umount lustre-MDT0001 complete [ 9343.964628] Lustre: server umount lustre-OST0000 complete [ 9357.311094] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9357.711514] LustreError: 339203:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9357.788715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9361.518549] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 9362.922445] LustreError: 339204:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9368.039751] LustreError: 339203:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9370.008966] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9370.573552] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9375.267329] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 9380.344852] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9383.968744] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9390.115495] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9390.463452] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9396.632663] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 9402.812688] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 9403.042165] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9416.673607] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9416.681880] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9416.710754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9421.849281] Lustre: server umount lustre-MDT0000 complete [ 9423.339435] LustreError: 339204:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9423.360289] LustreError: 339204:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9426.037367] LustreError: 339184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775855988 with bad export cookie 3855072152987928612 [ 9426.037795] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9426.049711] LustreError: 339184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9426.621603] Lustre: server umount lustre-MDT0001 complete [ 9442.155926] Lustre: server umount lustre-OST0000 complete [ 9458.942609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9459.504426] LustreError: 342784:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9459.578949] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9464.010471] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 9472.418635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9477.163788] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 9483.999336] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9489.812074] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9497.740541] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9498.163787] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9498.173536] Lustre: Skipped 1 previous similar message [ 9504.866496] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 9505.354699] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 9511.040321] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9518.054784] Lustre: server umount lustre-OST0000 complete [ 9523.170738] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9523.175955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9523.181369] Lustre: Skipped 3 previous similar messages [ 9523.208301] Lustre: Skipped 4 previous similar messages [ 9528.289380] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9528.300404] Lustre: Skipped 1 previous similar message [ 9532.525849] Lustre: server umount lustre-MDT0000 complete [ 9534.434367] LustreError: 342780:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9534.448448] LustreError: 342780:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9537.094962] LustreError: 342765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775856099 with bad export cookie 3855072152987929641 [ 9537.098635] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9537.106768] LustreError: 342765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9537.429597] Lustre: server umount lustre-MDT0001 complete [ 9549.492645] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 9553.409750] Key type lgssc unregistered [ 9553.880546] LNet: 345963:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9553.895913] LNetError: 345963:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9553.920633] LNet: Removed LNI 192.168.203.137@tcp [ 9555.010665] Key type .llcrypt unregistered [ 9555.014078] Key type ._llcrypt unregistered [ 9584.527264] Key type ._llcrypt registered [ 9584.530324] Key type .llcrypt registered [ 9584.666656] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 9585.971926] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9586.009421] alg: No test for adler32 (adler32-zlib) [ 9587.087920] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [ 9587.349099] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 9589.072242] Key type lgssc registered [ 9590.246688] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9598.512884] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9598.537479] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9598.551383] LustreError: 347391:0:(osd_handler.c:8532:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 9598.574540] LustreError: 347391:0:(obd_config.c:843:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 9598.587485] LustreError: 347391:0:(obd_mount.c:250:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 9598.596603] LustreError: 347391:0:(tgt_mount.c:2513:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 9598.610716] LustreError: 347391:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 9607.121397] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9607.134376] LustreError: 347788:0:(osd_handler.c:8532:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 9607.166223] LustreError: 347788:0:(obd_config.c:843:class_setup()) setup lustre-OST0000-osd failed (-22) [ 9607.181171] LustreError: 347788:0:(obd_mount.c:250:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 9607.188654] LustreError: 347788:0:(tgt_mount.c:2513:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 9607.201929] LustreError: 347788:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 9628.353895] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 9631.223495] Key type lgssc unregistered [ 9631.536789] LNet: 348774:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9631.548618] LNetError: 348774:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9631.567928] LNet: Removed LNI 192.168.203.137@tcp [ 9632.428787] Key type .llcrypt unregistered [ 9632.431498] Key type ._llcrypt unregistered [ 9652.239161] Key type ._llcrypt registered [ 9652.245519] Key type .llcrypt registered [ 9652.357847] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_hostid [ 9669.982495] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 9671.226797] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9671.263901] alg: No test for adler32 (adler32-zlib) [ 9672.406391] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [ 9672.707033] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 9674.514078] Key type lgssc registered [ 9675.632695] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9715.099523] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9715.129052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9716.561476] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9716.612177] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9716.711565] Lustre: lustre-MDT0000: new disk, initializing [ 9716.824704] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9716.870616] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9722.010768] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 9733.703211] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9733.831400] Lustre: 351913:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9733.916168] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9733.919864] Lustre: Skipped 1 previous similar message [ 9734.019492] Lustre: lustre-MDT0001: new disk, initializing [ 9734.155364] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9734.210877] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9734.229859] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9739.388051] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 9749.478956] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9754.102524] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9761.983719] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9762.316512] Lustre: lustre-OST0000: new disk, initializing [ 9762.323452] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9762.444654] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9769.457692] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 9770.603773] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9770.625525] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9770.750483] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9777.922072] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9782.854780] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9783.171529] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9790.318100] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9790.653652] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9796.585424] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9796.594320] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9796.598223] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9796.630826] Lustre: Skipped 1 previous similar message [ 9799.410847] Lustre: server umount lustre-OST0000 complete [ 9805.796361] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9805.804203] LustreError: Skipped 1 previous similar message [ 9805.815722] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9805.825330] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9805.833143] Lustre: Skipped 1 previous similar message [ 9806.817794] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9806.836847] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9806.843926] Lustre: Skipped 1 previous similar message [ 9810.503906] Lustre: server umount lustre-MDT0000 complete [ 9814.247513] LustreError: 351905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775856376 with bad export cookie 4647719327203007247 [ 9814.256491] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9814.264801] LustreError: 351905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9814.650410] Lustre: server umount lustre-MDT0001 complete [ 9824.001990] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 9825.798351] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 17:26:25 (1775856385) [ 9834.836866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9835.505970] LustreError: 355971:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9835.530777] LustreError: 355971:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9835.667798] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9841.130232] LustreError: 355972:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9841.626855] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 9846.245733] LustreError: 355971:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9851.260267] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9851.380650] LustreError: 355972:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9851.785322] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9856.879236] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 9863.928449] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9867.827631] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9875.938299] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9876.304944] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9883.092913] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 9889.512494] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9897.545571] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9897.701486] Lustre: lustre-OST0001: new disk, initializing [ 9897.708335] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9897.813942] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9903.189417] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9903.209092] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9903.296387] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9905.123870] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 9913.507637] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 9918.953248] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9918.958582] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9918.962386] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9918.984928] Lustre: Skipped 1 previous similar message [ 9919.030766] LustreError: Skipped 1 previous similar message [ 9924.062472] Lustre: server umount lustre-OST0001 complete [ 9924.067688] LustreError: 357568:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9924.104431] LustreError: 357568:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9934.334627] LustreError: 357568:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9934.362365] LustreError: 357568:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 9943.009178] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9943.024703] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9943.046834] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9943.061868] Lustre: Skipped 2 previous similar messages [ 9948.914175] Lustre: server umount lustre-OST0000 complete [ 9954.273438] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9954.298266] LustreError: Skipped 1 previous similar message [ 9954.310991] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9954.333080] Lustre: Skipped 1 previous similar message [ 9954.352714] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9954.370620] Lustre: Skipped 2 previous similar messages [ 9960.119769] Lustre: server umount lustre-MDT0000 complete [ 9964.544923] LustreError: 355952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775856527 with bad export cookie 4647719327203008122 [ 9964.563533] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9971.305139] Lustre: server umount lustre-MDT0001 complete [ 9985.112544] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 9988.908456] Key type lgssc unregistered [ 9989.342888] LNet: 360334:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9989.376290] LNetError: 360334:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9990.443639] LNet: Removed LNI 192.168.203.137@tcp [ 9991.627387] Key type .llcrypt unregistered [ 9991.634992] Key type ._llcrypt unregistered [10047.831505] Key type ._llcrypt registered [10047.837370] Key type .llcrypt registered [10047.975956] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [10048.959186] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10049.188904] alg: No test for adler32 (adler32-zlib) [10050.407502] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [10050.706559] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [10052.424224] Key type lgssc registered [10053.657674] Lustre: Echo OBD driver; http://www.lustre.org/ [10063.531304] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10063.543630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10064.772994] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10064.797790] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10064.966977] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10069.967336] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [10079.268981] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10079.425050] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10079.478242] Lustre: 363117:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10079.669247] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [10079.760292] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10084.650926] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [10090.785140] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10095.042249] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10100.195304] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10100.201326] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10100.202546] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10100.230235] Lustre: Skipped 1 previous similar message [10103.608502] Lustre: server umount lustre-MDT0000 complete [10108.109494] LustreError: 363110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775856670 with bad export cookie 16529790257919348523 [10108.115344] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10108.142667] LustreError: 363110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10108.729781] Lustre: server umount lustre-MDT0001 complete [10119.984761] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 17:31:20 (1775856680) [10127.109390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10127.610230] LustreError: 365402:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10127.631843] LustreError: 365402:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10127.668976] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10132.755855] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [10132.963995] LustreError: 365403:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10138.082651] LustreError: 365402:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10143.205682] LustreError: 365403:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10143.869402] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10144.413723] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10150.299529] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [10157.410301] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10161.838920] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10169.722346] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10170.010318] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10170.293255] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10177.202917] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [10182.718081] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10191.040695] Lustre: Permanently deactivating lustre-OST0000 [10191.057474] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [10199.108940] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10200.895939] LustreError: 368048:0:(mgs_llog.c:1942:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [10200.924761] LustreError: 368048:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [10200.940254] LustreError: 368048:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10204.993856] LustreError: 368097:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [10205.005381] LustreError: 368097:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10209.456769] Lustre: server umount lustre-OST0000 complete [10212.601321] LustreError: 368346:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [10212.607569] LustreError: 368346:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10215.906017] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10215.915287] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10215.937603] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10215.950199] Lustre: Skipped 1 previous similar message [10217.957254] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10217.960985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10218.015788] Lustre: Skipped 1 previous similar message [10223.074430] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10223.081160] Lustre: Skipped 1 previous similar message [10225.230213] Lustre: server umount lustre-MDT0000 complete [10228.193680] LustreError: 366606:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10228.216877] LustreError: 366606:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10229.396828] LustreError: 365384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775856791 with bad export cookie 16529790257919349104 [10229.398298] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10229.413794] LustreError: 365384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10239.137185] Lustre: server umount lustre-MDT0001 complete [10248.378732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10253.493987] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [10256.411541] LustreError: 369490:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [10263.634415] Lustre: 369829:0:(mgs_llog.c:1709:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.137@tcp [10265.809674] Lustre: server umount MGS complete [10278.666600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10279.113797] LustreError: 370553:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10279.130346] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10279.135767] Lustre: Skipped 1 previous similar message [10279.179255] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10283.522437] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [10292.461144] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10292.794615] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10297.326709] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [10303.145160] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10307.421954] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10309.876471] Lustre: Permanently reactivating lustre-OST0000 [10309.887397] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [10309.895519] Lustre: Skipped 2 previous similar messages [10336.225332] LustreError: 372062:0:(osp_dev.c:694:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [10345.052655] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10345.451475] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10345.455907] Lustre: Skipped 1 previous similar message [10351.868544] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [10358.545479] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10365.928650] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10365.943306] Lustre: Skipped 1 previous similar message [10365.944511] LustreError: 372936:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10365.977498] LustreError: 372936:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10366.124750] Lustre: server umount lustre-OST0000 complete [10371.044503] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10371.058953] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10371.070963] Lustre: Skipped 1 previous similar message [10384.352361] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10384.603697] Lustre: server umount lustre-MDT0000 complete [10389.613458] LustreError: 370534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775856952 with bad export cookie 16529790257919350147 [10389.614694] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10389.621932] LustreError: 370534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10390.210520] Lustre: server umount lustre-MDT0001 complete [10401.621668] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [10404.584911] Key type lgssc unregistered [10404.937496] LNet: 374519:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10404.943866] LNetError: 374519:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10404.961276] LNet: Removed LNI 192.168.203.137@tcp [10406.027888] Key type .llcrypt unregistered [10406.031108] Key type ._llcrypt unregistered [10425.943127] Key type ._llcrypt registered [10425.950629] Key type .llcrypt registered [10426.077561] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_hostid [10442.077564] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [10442.969794] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10443.005604] alg: No test for adler32 (adler32-zlib) [10444.068746] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [10444.282587] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [10446.040683] Key type lgssc registered [10446.996246] Lustre: Echo OBD driver; http://www.lustre.org/ [10482.986766] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10483.038723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10484.370236] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10484.434320] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10484.553875] Lustre: lustre-MDT0000: new disk, initializing [10484.783850] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10484.848659] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10489.393752] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [10501.578883] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10501.694329] Lustre: 377659:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10501.748160] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10501.759675] Lustre: Skipped 1 previous similar message [10501.918593] Lustre: lustre-MDT0001: new disk, initializing [10502.091827] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10502.154370] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10502.184475] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10507.683141] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [10517.529327] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10522.079632] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10529.000860] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10529.329160] Lustre: lustre-OST0000: new disk, initializing [10529.333745] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10529.422538] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10536.022396] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10536.043389] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10536.218228] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10537.388714] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [10545.754606] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10550.340296] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10550.532249] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10554.839188] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10555.009385] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10556.897535] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10556.906056] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10556.915485] Lustre: Skipped 1 previous similar message [10556.916388] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10556.931197] Lustre: Skipped 1 previous similar message [10562.026589] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10563.068429] Lustre: server umount lustre-OST0000 complete [10567.140468] LustreError: 375881:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000e14aefcd x1862121035963264/t0(0) o8->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10567.145473] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10567.180280] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10567.195604] Lustre: Skipped 1 previous similar message [10567.218816] Lustre: Skipped 2 previous similar messages [10572.195710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10572.209174] Lustre: Skipped 3 previous similar messages [10573.556718] Lustre: server umount lustre-MDT0000 complete [10576.789784] LustreError: 377651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775857139 with bad export cookie 12925889552798815917 [10576.802562] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10576.803751] LustreError: 377651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10577.079571] Lustre: server umount lustre-MDT0001 complete [10585.642871] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [10587.419942] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 17:39:07 (1775857147) [10594.719968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10595.522069] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10599.927611] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [10600.938296] LustreError: 381717:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10600.972560] LustreError: 381717:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10605.029457] LustreError: 381718:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10609.976307] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10610.147341] LustreError: 381717:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10610.458473] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10616.163614] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [10621.749412] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10625.649489] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10632.845414] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10633.213776] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10639.444875] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [10645.896638] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10647.981560] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [10652.099646] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [10660.839547] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10660.845654] LustreError: Skipped 1 previous similar message [10660.848560] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10660.858430] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10660.863399] Lustre: Skipped 1 previous similar message [10663.492562] Lustre: server umount lustre-OST0000 complete [10671.082860] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10671.093133] Lustre: Skipped 1 previous similar message [10671.094584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10671.108108] Lustre: Skipped 1 previous similar message [10673.433711] Lustre: server umount lustre-MDT0000 complete [10676.200475] LustreError: 381713:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10676.222696] LustreError: 381713:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10676.990440] LustreError: 384358:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775857239 with bad export cookie 12925889552798816799 [10677.006105] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10677.007041] LustreError: 384358:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10677.334409] Lustre: server umount lustre-MDT0001 complete [10687.798428] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [10690.295483] Key type lgssc unregistered [10690.506091] LNet: 385093:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10690.514623] LNetError: 385093:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10690.535258] LNet: Removed LNI 192.168.203.137@tcp [10691.292418] Key type .llcrypt unregistered [10691.298098] Key type ._llcrypt unregistered [10714.664371] Key type ._llcrypt registered [10714.671201] Key type .llcrypt registered [10714.776939] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [10715.014567] Key type .llcrypt unregistered [10715.017571] Key type ._llcrypt unregistered [10730.311371] Key type ._llcrypt registered [10730.314976] Key type .llcrypt registered [10730.483587] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [10731.655738] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10731.708704] alg: No test for adler32 (adler32-zlib) [10732.742577] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [10733.005556] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [10734.704167] Key type lgssc registered [10736.025728] Lustre: Echo OBD driver; http://www.lustre.org/ [10743.826711] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10743.858837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10745.343401] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10749.332460] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [10754.536557] LustreError: 387271:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10754.548977] LustreError: 387271:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10757.528678] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10757.862581] LustreError: 387270:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10757.890689] LustreError: 387270:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10757.915915] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10763.241991] LustreError: 387271:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10763.256790] LustreError: 387271:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10765.008640] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [10768.354693] LustreError: 387270:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10768.375379] LustreError: 387270:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10770.439801] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10771.233263] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10775.538892] LustreError: 387271:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10775.563536] LustreError: 387271:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10777.712472] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10778.128633] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10781.804744] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [10790.883863] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10790.900671] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10790.914909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10791.211445] Lustre: server umount lustre-OST0000 complete [10797.045437] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [10797.057896] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10797.086552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10798.562773] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10798.570321] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10798.605279] Lustre: Skipped 1 previous similar message [10800.996035] Lustre: server umount lustre-MDT0000 complete [10803.692390] LustreError: 387270:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10803.705577] LustreError: 387270:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10804.353399] LustreError: 387253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775857366 with bad export cookie 12251290671742356926 [10804.353710] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10804.376076] LustreError: 387253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10814.147773] Lustre: server umount lustre-MDT0001 complete [10824.513804] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [10826.939890] Key type lgssc unregistered [10827.189733] LNet: 390257:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10827.196892] LNetError: 390257:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10827.214064] LNet: Removed LNI 192.168.203.137@tcp [10827.984118] Key type .llcrypt unregistered [10827.985415] Key type ._llcrypt unregistered [10850.665027] Key type ._llcrypt registered [10850.668821] Key type .llcrypt registered [10850.782098] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [10851.753567] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10851.866583] alg: No test for adler32 (adler32-zlib) [10852.937405] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [10853.148890] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [10854.800260] Key type lgssc registered [10856.032304] Lustre: Echo OBD driver; http://www.lustre.org/ [10863.329167] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10863.346642] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10878.944360] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [10884.064189] LustreError: 391631:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.137@tcp: failed processing log, type 4: rc = -110 [10910.752189] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10917.139938] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [10922.679617] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10929.308475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10929.629845] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10929.683788] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10933.748875] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [10934.767622] LustreError: 391656:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10934.784365] LustreError: 391656:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10939.876623] LustreError: 391657:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10942.551145] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10943.004698] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10947.062868] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [10952.156607] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10955.535424] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10963.430848] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10963.437611] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10963.455592] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10964.184716] Lustre: server umount lustre-OST0000 complete [10968.545402] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10968.551750] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10968.570828] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10971.617713] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10971.620438] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10971.643984] Lustre: Skipped 1 previous similar message [10976.737740] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10976.741380] Lustre: Skipped 1 previous similar message [10976.843727] Lustre: server umount lustre-MDT0000 complete [10979.878550] LustreError: 393827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775857542 with bad export cookie 3646176067433540915 [10979.883210] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10979.892353] LustreError: 393827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10980.158674] Lustre: server umount lustre-MDT0001 complete [10989.038625] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [10991.889243] Key type lgssc unregistered [10992.248959] LNet: 394849:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10992.257178] LNetError: 394849:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10992.271861] LNet: Removed LNI 192.168.203.137@tcp [10993.044884] Key type .llcrypt unregistered [10993.047606] Key type ._llcrypt unregistered [11017.416569] Key type ._llcrypt registered [11017.420183] Key type .llcrypt registered [11017.558392] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [11018.692187] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11018.714986] alg: No test for adler32 (adler32-zlib) [11019.844751] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [11020.100891] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [11021.817124] Key type lgssc registered [11022.968140] Lustre: Echo OBD driver; http://www.lustre.org/ [11030.575917] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11030.605283] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11032.101642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11036.209808] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [11042.279368] LustreError: 396248:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11042.303615] LustreError: 396248:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11044.491795] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11044.877840] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11049.161337] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [11054.421239] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11058.079797] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11064.463836] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11064.868484] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11070.028731] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [11075.114332] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11077.170028] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11085.794093] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11085.802284] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11085.815372] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11087.022787] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.37@tcp (stopping) [11087.346359] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11087.369791] Lustre: Skipped 1 previous similar message [11090.395696] Lustre: server umount lustre-MDT0000 complete [11092.119909] LustreError: 396242:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11092.751195] LustreError: 396230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.37@tcp arrived at 1775857655 with bad export cookie 9145904700897501031 [11092.758516] LustreError: 396230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11094.420802] LustreError: 396229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775857656 with bad export cookie 9145904700897500366 [11094.431244] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11095.522485] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11095.535188] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11095.549651] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11095.554521] Lustre: Skipped 3 previous similar messages [11097.569673] LustreError: 396247:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11097.569784] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11097.593452] LustreError: 396247:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [11099.617839] LustreError: 398206:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11099.640880] LustreError: 398206:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11100.736812] Lustre: server umount lustre-OST0000 complete [11103.713609] LustreError: 396247:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11103.735169] LustreError: 396247:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11106.715041] Lustre: server umount lustre-MDT0001 complete [11116.493731] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [11118.941854] Key type lgssc unregistered [11119.289780] LNet: 399580:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11119.305994] LNetError: 399580:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11119.324315] LNet: Removed LNI 192.168.203.137@tcp [11119.964769] Key type .llcrypt unregistered [11119.967079] Key type ._llcrypt unregistered [11142.293557] Key type ._llcrypt registered [11142.299166] Key type .llcrypt registered [11142.394455] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [11143.058939] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11143.186533] alg: No test for adler32 (adler32-zlib) [11144.228939] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [11144.422169] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [11146.112416] Key type lgssc registered [11147.209876] Lustre: Echo OBD driver; http://www.lustre.org/ [11154.481481] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11154.492330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11156.025757] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11160.075475] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [11166.183712] LustreError: 400980:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11166.200209] LustreError: 400980:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11167.577987] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11167.864336] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11171.554173] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [11177.362438] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11181.385425] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11187.989192] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11188.343149] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11193.499302] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [11198.260484] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11200.703791] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11206.115702] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11206.117363] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11206.124109] Lustre: Skipped 1 previous similar message [11209.379567] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.37@tcp (stopping) [11209.385687] Lustre: Skipped 1 previous similar message [11211.233860] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11211.240053] Lustre: Skipped 1 previous similar message [11212.202254] Lustre: server umount lustre-MDT0001 complete [11216.355430] LustreError: 400980:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11216.373425] LustreError: 400980:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11221.473273] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11221.475319] LustreError: 400981:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11221.480066] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11221.508102] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11223.057778] Lustre: server umount lustre-OST0000 complete [11226.594092] LustreError: 400980:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11227.137361] Lustre: server umount lustre-MDT0000 complete [11239.326791] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [11241.811595] Key type lgssc unregistered [11242.067062] LNet: 404313:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11242.076515] LNetError: 404313:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11242.102760] LNet: Removed LNI 192.168.203.137@tcp [11242.884768] Key type .llcrypt unregistered [11242.888063] Key type ._llcrypt unregistered [11267.040936] Key type ._llcrypt registered [11267.045905] Key type .llcrypt registered [11267.161471] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_hostid [11281.907748] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [11282.680274] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11282.699681] alg: No test for adler32 (adler32-zlib) [11283.815603] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [11284.020478] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [11285.680162] Key type lgssc registered [11286.790636] Lustre: Echo OBD driver; http://www.lustre.org/ [11323.197532] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [11338.408094] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11338.453902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11339.797358] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11339.861663] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11339.926871] Lustre: lustre-MDT0000: new disk, initializing [11340.012923] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11340.027415] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11344.034825] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [11358.204353] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11358.424369] Lustre: lustre-OST0000: new disk, initializing [11358.428276] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11358.434742] Lustre: Skipped 1 previous similar message [11358.498775] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11364.176774] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [11366.001392] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [11366.012221] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [11366.044047] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [11371.333769] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11385.907655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11386.052469] Lustre: 408277:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11386.122753] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11386.195320] Lustre: lustre-MDT0001: new disk, initializing [11386.316493] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11386.353805] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [11386.372131] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [11390.224606] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [11399.456416] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [11399.706074] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [11404.034168] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [11404.250573] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [11411.426260] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [11411.432342] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11411.449839] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11411.460280] Lustre: Skipped 1 previous similar message [11412.657820] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.37@tcp (stopping) [11412.669061] Lustre: Skipped 1 previous similar message [11414.719119] Lustre: server umount lustre-MDT0001 complete [11421.679824] LustreError: 408285:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11421.724098] LustreError: 408285:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [11426.788311] LustreError: 409999:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11426.825314] LustreError: 409999:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11431.908762] LustreError: 408288:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11431.945289] LustreError: 408288:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11437.035572] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11437.036491] LustreError: 408289:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11437.047073] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11442.146250] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11442.146979] LustreError: 408288:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11447.265815] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11447.266684] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 4. Is it stuck? [11447.526590] Lustre: server umount lustre-OST0000 complete [11465.184258] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11465.620693] Lustre: server umount lustre-MDT0000 complete [11478.238435] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [11480.629433] Key type lgssc unregistered [11480.869610] LNet: 412848:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11480.875681] LNetError: 412848:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11480.887839] LNet: Removed LNI 192.168.203.137@tcp [11481.560682] Key type .llcrypt unregistered [11481.564972] Key type ._llcrypt unregistered [11519.720952] Key type ._llcrypt registered [11519.723780] Key type .llcrypt registered [11519.838090] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [11520.228646] Key type .llcrypt unregistered [11520.230705] Key type ._llcrypt unregistered [11535.497146] Key type ._llcrypt registered [11535.499742] Key type .llcrypt registered [11535.607041] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [11536.926090] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11536.947682] alg: No test for adler32 (adler32-zlib) [11538.011703] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [11538.323474] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [11540.000326] Key type lgssc registered [11541.043076] Lustre: Echo OBD driver; http://www.lustre.org/ [11568.622546] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11568.644165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11570.010296] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11570.087730] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11570.188266] Lustre: lustre-MDT0000: new disk, initializing [11570.354767] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11570.374764] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11575.188566] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [11588.466959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11588.701779] Lustre: 416193:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11588.827215] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11588.846037] Lustre: Skipped 1 previous similar message [11588.988922] Lustre: lustre-MDT0001: new disk, initializing [11589.170970] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11589.248245] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11589.280869] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11594.926501] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [11602.494985] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11606.769797] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11613.474517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11613.834256] Lustre: lustre-OST0000: new disk, initializing [11613.843396] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11614.005290] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11619.429267] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11619.452757] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11619.558469] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11620.262397] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [11628.234977] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11635.176068] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11635.189497] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11635.191065] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11635.197074] Lustre: Skipped 1 previous similar message [11640.292202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11640.303205] Lustre: Skipped 2 previous similar messages [11640.971609] Lustre: server umount lustre-MDT0000 complete [11644.611357] LustreError: 416184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775858207 with bad export cookie 13322873772683383616 [11644.626992] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11644.950840] Lustre: server umount lustre-MDT0001 complete [11659.090892] Lustre: server umount lustre-OST0000 complete [11669.762043] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 17:57:09 (1775858229) [11678.499573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11679.086782] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11683.128325] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [11684.325458] LustreError: 419850:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11684.348784] LustreError: 419850:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11689.453045] LustreError: 419849:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11691.093507] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11691.328812] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11695.202387] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [11700.646917] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11704.354191] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11724.454780] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11724.617992] Lustre: Found index 0 for lustre-OST0000, updating log [11724.756918] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11730.042722] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [11737.551852] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11749.381510] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [11761.636526] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11761.643737] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11761.654451] Lustre: Skipped 2 previous similar messages [11761.669757] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11766.332557] Lustre: server umount lustre-OST0000 complete [11783.653959] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11783.671867] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11783.700205] Lustre: Skipped 1 previous similar message [11783.710038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11783.720064] Lustre: Skipped 3 previous similar messages [11788.217865] Lustre: server umount lustre-MDT0000 complete [11789.221155] LNet: 1 local NIs in recovery (showing 1): 192.168.203.137@tcp [11789.285036] LustreError: 419849:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11789.298388] LustreError: 419849:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11792.078737] LustreError: 419830:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775858354 with bad export cookie 13322873772683385009 [11792.081481] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11792.087164] LustreError: 419830:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [11792.478243] Lustre: server umount lustre-MDT0001 complete [11803.045165] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 17:59:22 (1775858362) [12425.611810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12426.071802] LustreError: 425955:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12426.099103] LustreError: 425955:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12426.176527] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12430.655119] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [12431.334324] LustreError: 425956:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12436.453991] LustreError: 425955:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12440.330499] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12440.983219] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12445.700723] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [12452.260900] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12456.609677] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12477.244210] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12477.505122] Lustre: Found index 0 for lustre-OST0000, updating log [12477.805388] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12480.482459] LNet: 1 local NIs in recovery (showing 1): 192.168.203.137@tcp [12484.237515] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [12490.346043] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12493.283561] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12493.303525] Lustre: Skipped 1 previous similar message [12493.310939] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12493.323886] Lustre: Skipped 2 previous similar messages [12495.169884] Lustre: server umount lustre-OST0000 complete [12500.516087] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12500.536499] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12500.555071] Lustre: Skipped 1 previous similar message [12500.592517] Lustre: Skipped 1 previous similar message [12501.606403] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12501.616240] Lustre: Skipped 1 previous similar message [12503.650581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12503.663177] Lustre: Skipped 2 previous similar messages [12505.978878] Lustre: server umount lustre-MDT0000 complete [12506.739470] LustreError: 425956:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12506.767247] LustreError: 425956:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12510.691105] LustreError: 425937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775859073 with bad export cookie 13322873772683386094 [12510.706518] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12510.716349] LustreError: 425937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12511.024196] Lustre: server umount lustre-MDT0001 complete [12540.899049] LNet: 1 local NIs in recovery (showing 1): 192.168.203.137@tcp [12545.844904] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_hostid [12558.265750] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [12596.062079] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 18:12:36 (1775859156) [12613.111127] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [12615.437464] Key type lgssc unregistered [12615.776474] LNet: 432183:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12615.785289] LNetError: 432183:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12739.686920] LNet: Removed LNI 192.168.203.137@tcp [12741.080573] Key type .llcrypt unregistered [12741.082444] Key type ._llcrypt unregistered [12762.812701] Key type ._llcrypt registered [12762.822507] Key type .llcrypt registered [12763.076332] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [12764.727584] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12765.000082] alg: No test for adler32 (adler32-zlib) [12766.611336] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [12766.990615] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [12768.720658] Key type lgssc registered [12769.793463] Lustre: Echo OBD driver; http://www.lustre.org/ [12775.118921] LNet: Added LNI 192.168.203.137@tcp100 [8/256/0/180] [12776.083040] LNet: Added LNI 192.168.203.137@tcp101 [8/256/0/180] [12777.080662] LNet: Added LNI 192.168.203.137@tcp102 [8/256/0/180] [12777.905355] LNet: Added LNI 192.168.203.137@tcp103 [8/256/0/180] [12778.779550] LNet: Added LNI 192.168.203.137@tcp104 [8/256/0/180] [12779.658305] LNet: Added LNI 192.168.203.137@tcp105 [8/256/0/180] [12780.569525] LNet: Added LNI 192.168.203.137@tcp106 [8/256/0/180] [12781.340155] LNet: Added LNI 192.168.203.137@tcp107 [8/256/0/180] [12782.290291] LNet: Added LNI 192.168.203.137@tcp108 [8/256/0/180] [12783.230395] LNet: Added LNI 192.168.203.137@tcp109 [8/256/0/180] [12784.083425] LNet: Added LNI 192.168.203.137@tcp110 [8/256/0/180] [12784.913506] LNet: Added LNI 192.168.203.137@tcp111 [8/256/0/180] [12785.833106] LNet: Added LNI 192.168.203.137@tcp112 [8/256/0/180] [12786.659496] LNet: Added LNI 192.168.203.137@tcp113 [8/256/0/180] [12787.515176] LNet: Added LNI 192.168.203.137@tcp114 [8/256/0/180] [12788.318619] LNet: Added LNI 192.168.203.137@tcp115 [8/256/0/180] [12789.280122] LNet: Added LNI 192.168.203.137@tcp116 [8/256/0/180] [12790.198417] LNet: Added LNI 192.168.203.137@tcp117 [8/256/0/180] [12791.224455] LNet: Added LNI 192.168.203.137@tcp118 [8/256/0/180] [12792.143067] LNet: Added LNI 192.168.203.137@tcp119 [8/256/0/180] [12793.195042] LNet: Added LNI 192.168.203.137@tcp120 [8/256/0/180] [12794.196189] LNet: Added LNI 192.168.203.137@tcp121 [8/256/0/180] [12795.119367] LNet: Added LNI 192.168.203.137@tcp122 [8/256/0/180] [12796.197617] LNet: Added LNI 192.168.203.137@tcp123 [8/256/0/180] [12797.084668] LNet: Added LNI 192.168.203.137@tcp124 [8/256/0/180] [12798.174121] LNet: Added LNI 192.168.203.137@tcp125 [8/256/0/180] [12799.179624] LNet: Added LNI 192.168.203.137@tcp126 [8/256/0/180] [12800.129800] LNet: Added LNI 192.168.203.137@tcp127 [8/256/0/180] [12801.033262] LNet: Added LNI 192.168.203.137@tcp128 [8/256/0/180] [12802.119689] LNet: Added LNI 192.168.203.137@tcp129 [8/256/0/180] [12803.210628] LNet: Added LNI 192.168.203.137@tcp130 [8/256/0/180] [12804.288778] LNet: Added LNI 192.168.203.137@tcp131 [8/256/0/180] [12805.445475] LNet: Added LNI 192.168.203.137@tcp132 [8/256/0/180] [12806.513444] LNet: Added LNI 192.168.203.137@tcp133 [8/256/0/180] [12807.643521] LNet: Added LNI 192.168.203.137@tcp134 [8/256/0/180] [12808.465625] LNet: Added LNI 192.168.203.137@tcp135 [8/256/0/180] [12816.287045] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12816.320930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12817.490589] Lustre: 435204:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.137@tcp: can't add NID 192.168.203.137@tcp130: rc = -75 [12817.576097] LNetError: 432664:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.137 [12817.602279] LNetError: Protocol error connecting to 192.168.203.137@tcp121 at host 192.168.203.137:988: is it running a compatible version of Lustre? [12817.794373] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12817.862557] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12818.007770] Lustre: lustre-MDT0000: new disk, initializing [12818.093564] Lustre: 435259:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.137@tcp: can't add NID 192.168.203.137@tcp130: rc = -75 [12818.099538] Lustre: 435259:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [12818.120484] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12818.135910] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12818.220661] LNet: 1 local NIs in recovery (showing 1): 192.168.203.137@tcp121 [12818.231913] LNetError: 432661:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.137 [12818.231972] LNetError: Protocol error connecting to 192.168.203.137@tcp112 at host 192.168.203.137:988: is it running a compatible version of Lustre? [12820.323799] LNetError: 432661:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.137 [12820.343732] LNetError: 432661:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Skipped 4 previous similar messages [12820.355600] LNetError: Protocol error connecting to 192.168.203.137@tcp121 at host 192.168.203.137:988: is it running a compatible version of Lustre? [12820.371556] LNetError: Skipped 4 previous similar messages [12823.336586] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [12824.421621] LNetError: 432662:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.137 [12824.451592] LNetError: Protocol error connecting to 192.168.203.137@tcp121 at host 192.168.203.137:988: is it running a compatible version of Lustre? [12832.625129] LNetError: 435226:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.137 [12832.662675] LNetError: Protocol error connecting to 192.168.203.137@tcp121 at host 192.168.203.137:988: is it running a compatible version of Lustre? [12836.511760] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12836.622294] Lustre: 435221:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12836.692868] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12836.698189] Lustre: Skipped 1 previous similar message [12836.772757] Lustre: lustre-MDT0001: new disk, initializing [12836.800852] Lustre: 436006:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.137@tcp: can't add NID 192.168.203.137@tcp130: rc = -75 [12836.807662] Lustre: 436006:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [12836.923715] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12836.983280] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12836.999523] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12842.514680] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [12843.601868] Lustre: 436286:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.137@tcp: can't add NID 192.168.203.137@tcp130: rc = -75 [12843.630668] Lustre: 436286:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [12849.011236] LNetError: 435225:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.137 [12849.028960] LNetError: Protocol error connecting to 192.168.203.137@tcp121 at host 192.168.203.137:988: is it running a compatible version of Lustre? [12853.187856] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12858.776750] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12865.314781] LNetError: 432661:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.137 [12865.337641] LNetError: 432661:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Skipped 5 previous similar messages [12865.361830] LNetError: Protocol error connecting to 192.168.203.137@tcp114 at host 192.168.203.137:988: is it running a compatible version of Lustre? [12865.379987] LNetError: Skipped 5 previous similar messages [12867.587482] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12868.050299] Lustre: lustre-OST0000: new disk, initializing [12868.068807] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12868.259877] Lustre: 436974:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.137@tcp: can't add NID 192.168.203.137@tcp130: rc = -75 [12868.279479] Lustre: 436974:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [12868.338916] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12873.824463] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12873.831764] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12873.992725] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12875.144830] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [12877.664180] LNet: 1 local NIs in recovery (showing 1): 192.168.203.137@tcp121 [12883.520620] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12894.691748] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12894.696614] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12894.706886] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12894.727757] Lustre: Skipped 1 previous similar message [12898.183762] Lustre: server umount lustre-OST0000 complete [12901.217160] LNetError: 435227:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.137 [12901.240362] LNetError: 435227:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Skipped 6 previous similar messages [12901.249489] LNetError: Protocol error connecting to 192.168.203.137@tcp111 at host 192.168.203.137:988: is it running a compatible version of Lustre? [12901.262604] LNetError: Skipped 6 previous similar messages [12903.394693] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12903.403787] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12903.421179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12903.428622] Lustre: Skipped 1 previous similar message [12904.930376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12904.931218] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12908.348467] Lustre: server umount lustre-MDT0000 complete [12912.137698] LustreError: 435212:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775859474 with bad export cookie 1212151309376218000 [12912.153592] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12912.516946] Lustre: server umount lustre-MDT0001 complete [12924.438239] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [12927.080790] Key type lgssc unregistered [12927.427186] LNet: 438752:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12927.438261] LNetError: 438752:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12927.446747] LNet: Removed LNI 192.168.203.137@tcp [12928.480896] LNet: Removed LNI 192.168.203.137@tcp100 [12928.487679] LNetError: 438752:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12928.498345] LNetError: 438752:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [12928.515715] LNet: Removed LNI 192.168.203.137@tcp101 [12929.569894] LNet: Removed LNI 192.168.203.137@tcp102 [12929.575172] LNetError: 438752:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12929.586914] LNetError: 438752:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [12929.597433] LNet: Removed LNI 192.168.203.137@tcp103 [12930.656329] LNet: Removed LNI 192.168.203.137@tcp104 [12931.680332] LNet: Removed LNI 192.168.203.137@tcp105 [12931.686974] LNetError: 438752:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12931.694365] LNetError: 438752:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [12931.700054] LNet: Removed LNI 192.168.203.137@tcp106 [12932.704282] LNet: Removed LNI 192.168.203.137@tcp107 [12933.728174] LNet: Removed LNI 192.168.203.137@tcp108 [12934.752535] LNet: Removed LNI 192.168.203.137@tcp109 [12935.776299] LNet: Removed LNI 192.168.203.137@tcp110 [12935.780931] LNetError: 438752:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12935.787415] LNetError: 438752:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [12935.799826] LNet: Removed LNI 192.168.203.137@tcp111 [12936.864456] LNet: Removed LNI 192.168.203.137@tcp112 [12937.888217] LNet: Removed LNI 192.168.203.137@tcp113 [12938.912254] LNet: Removed LNI 192.168.203.137@tcp114 [12939.936943] LNet: Removed LNI 192.168.203.137@tcp115 [12940.961804] LNet: Removed LNI 192.168.203.137@tcp116 [12941.984221] LNet: Removed LNI 192.168.203.137@tcp117 [12943.008938] LNet: Removed LNI 192.168.203.137@tcp118 [12944.032393] LNet: Removed LNI 192.168.203.137@tcp119 [12944.039252] LNetError: 438752:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12944.051321] LNetError: 438752:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [12945.120171] LNet: Removed LNI 192.168.203.137@tcp120 [12945.122514] LNet: Removed LNI 192.168.203.137@tcp121 [12945.126144] LNet: Removed LNI 192.168.203.137@tcp122 [12945.129575] LNet: Removed LNI 192.168.203.137@tcp123 [12945.133543] LNet: Removed LNI 192.168.203.137@tcp124 [12945.137318] LNet: Removed LNI 192.168.203.137@tcp125 [12945.140812] LNet: Removed LNI 192.168.203.137@tcp126 [12945.144350] LNet: Removed LNI 192.168.203.137@tcp127 [12945.147571] LNet: Removed LNI 192.168.203.137@tcp128 [12945.158579] LNet: Removed LNI 192.168.203.137@tcp129 [12945.163671] LNet: Removed LNI 192.168.203.137@tcp130 [12945.169932] LNet: Removed LNI 192.168.203.137@tcp131 [12945.175180] LNet: Removed LNI 192.168.203.137@tcp132 [12945.180509] LNet: Removed LNI 192.168.203.137@tcp133 [12945.187940] LNet: Removed LNI 192.168.203.137@tcp134 [12945.198572] LNet: Removed LNI 192.168.203.137@tcp135 [12946.081749] Key type .llcrypt unregistered [12946.084533] Key type ._llcrypt unregistered [12971.372413] Key type ._llcrypt registered [12971.374561] Key type .llcrypt registered [12971.512389] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_hostid [12987.224883] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [12988.476214] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12988.500093] alg: No test for adler32 (adler32-zlib) [12989.596868] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [12989.854599] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [12991.568187] Key type lgssc registered [12992.853786] Lustre: Echo OBD driver; http://www.lustre.org/ [13028.133369] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 18:19:48 (1775859588) [13047.851302] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [13050.601217] Key type lgssc unregistered [13050.842805] LNet: 442787:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13050.849308] LNetError: 442787:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13050.862294] LNet: Removed LNI 192.168.203.137@tcp [13051.636592] Key type .llcrypt unregistered [13051.639961] Key type ._llcrypt unregistered [13069.071254] Key type ._llcrypt registered [13069.075545] Key type .llcrypt registered [13069.230379] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [13070.418179] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13070.606989] alg: No test for adler32 (adler32-zlib) [13071.743258] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [13072.024744] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [13073.856165] Key type lgssc registered [13075.117865] Lustre: Echo OBD driver; http://www.lustre.org/ [13084.794098] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13084.863442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13086.130236] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13086.160612] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13086.225063] Lustre: lustre-MDT0000: new disk, initializing [13086.291767] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13086.309826] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13090.292774] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [13101.486334] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13101.635447] Lustre: 443985:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13101.698085] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13101.703460] Lustre: Skipped 1 previous similar message [13101.829662] Lustre: lustre-MDT0001: new disk, initializing [13101.981136] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13102.061869] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13102.078259] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13107.314939] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [13116.682419] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13121.769946] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13129.269234] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13129.705750] Lustre: lustre-OST0000: new disk, initializing [13129.711605] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13129.835606] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13136.310621] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [13138.511688] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13138.533528] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13138.628806] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13144.087973] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13148.979487] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing lnet_if_list [13155.044593] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [13155.082969] LNet: Added LNI 192.168.203.137@tcp10 [8/256/0/180] [13160.804187] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [13160.866103] LNet: Added LNI 192.168.203.137@tcp11 [8/256/0/180] [13165.644579] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [13165.687675] LNet: Added LNI 192.168.203.137@tcp12 [8/256/0/180] [13170.445450] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [13170.480553] LNet: Added LNI 192.168.203.137@tcp13 [8/256/0/180] [13174.636298] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [13174.665147] LNet: Added LNI 192.168.203.137@tcp14 [8/256/0/180] [13179.215409] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [13179.276504] LNet: Added LNI 192.168.203.137@tcp15 [8/256/0/180] [13183.684740] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [13183.719454] LNet: Added LNI 192.168.203.137@tcp16 [8/256/0/180] [13188.137684] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [13188.174542] LNet: Added LNI 192.168.203.137@tcp17 [8/256/0/180] [13193.567683] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [13193.597704] LNet: Added LNI 192.168.203.137@tcp18 [8/256/0/180] [13198.640819] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [13198.686973] LNet: Added LNI 192.168.203.137@tcp19 [8/256/0/180] [13204.145968] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [13204.199374] LNet: Added LNI 192.168.203.137@tcp20 [8/256/0/180] [13209.179815] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [13209.215487] LNet: Added LNI 192.168.203.137@tcp21 [8/256/0/180] [13214.289928] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [13214.310304] LNet: Added LNI 192.168.203.137@tcp22 [8/256/0/180] [13219.363476] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [13219.388247] LNet: Added LNI 192.168.203.137@tcp23 [8/256/0/180] [13224.775728] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [13224.825735] LNet: Added LNI 192.168.203.137@tcp24 [8/256/0/180] [13230.662804] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [13230.706036] LNet: Added LNI 192.168.203.137@tcp25 [8/256/0/180] [13237.496974] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [13237.533053] LNet: Added LNI 192.168.203.137@tcp26 [8/256/0/180] [13244.890385] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [13244.967862] LNet: Added LNI 192.168.203.137@tcp27 [8/256/0/180] [13252.435387] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [13252.478610] LNet: Added LNI 192.168.203.137@tcp28 [8/256/0/180] [13258.443428] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [13258.518583] LNet: Added LNI 192.168.203.137@tcp29 [8/256/0/180] [13265.607225] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [13265.642849] LNet: Added LNI 192.168.203.137@tcp30 [8/256/0/180] [13272.423263] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [13272.481884] LNet: Added LNI 192.168.203.137@tcp31 [8/256/0/180] [13279.766420] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [13279.830828] LNet: Added LNI 192.168.203.137@tcp32 [8/256/0/180] [13288.704408] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [13288.760824] LNet: Added LNI 192.168.203.137@tcp33 [8/256/0/180] [13298.746991] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [13298.778957] LNet: Added LNI 192.168.203.137@tcp34 [8/256/0/180] [13309.324331] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [13309.366279] LNet: Added LNI 192.168.203.137@tcp35 [8/256/0/180] [13318.340100] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [13318.378499] LNet: Added LNI 192.168.203.137@tcp36 [8/256/0/180] [13325.663139] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [13325.691298] LNet: Added LNI 192.168.203.137@tcp37 [8/256/0/180] [13333.137023] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [13333.188291] LNet: Added LNI 192.168.203.137@tcp38 [8/256/0/180] [13342.816580] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [13342.853612] LNet: Added LNI 192.168.203.137@tcp39 [8/256/0/180] [13353.202492] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [13353.230213] LNet: Added LNI 192.168.203.137@tcp40 [8/256/0/180] [13361.174747] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [13361.225829] LNet: Added LNI 192.168.203.137@tcp41 [8/256/0/180] [13370.537660] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [13370.590582] LNet: Added LNI 192.168.203.137@tcp42 [8/256/0/180] [13380.024377] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [13380.062782] LNet: Added LNI 192.168.203.137@tcp43 [8/256/0/180] [13388.003929] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [13388.028114] LNet: Added LNI 192.168.203.137@tcp44 [8/256/0/180] [13396.225294] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [13396.266953] LNet: Added LNI 192.168.203.137@tcp45 [8/256/0/180] [13404.622796] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [13404.670829] LNet: Added LNI 192.168.203.137@tcp46 [8/256/0/180] [13412.631866] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [13412.665142] LNet: Added LNI 192.168.203.137@tcp47 [8/256/0/180] [13420.650483] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [13420.686742] LNet: Added LNI 192.168.203.137@tcp48 [8/256/0/180] [13427.791120] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [13427.834442] LNet: Added LNI 192.168.203.137@tcp49 [8/256/0/180] [13434.891854] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [13434.917872] LNet: Added LNI 192.168.203.137@tcp50 [8/256/0/180] [13450.211195] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13450.221059] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13450.279805] Lustre: server umount lustre-MDT0000 complete [13451.235721] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13452.697381] LustreError: 443977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775860015 with bad export cookie 5853816486335472904 [13452.698775] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13452.710716] LustreError: 443977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13452.948322] Lustre: server umount lustre-MDT0001 complete [13466.536342] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [13471.721156] Lustre: 443256:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775860018/real 1775860018] req@00000000d771bb04 x1862123791754240/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1775860034 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13471.743417] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13471.761287] Lustre: Skipped 1 previous similar message [13472.551443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13472.739784] Lustre: 443257:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775860019/real 1775860019] req@00000000e0589883 x1862123791754368/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1775860035 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13476.833515] Lustre: 443255:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775860023/real 1775860023] req@000000006ce57545 x1862123791754496/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1775860039 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13478.152747] LustreError: 445724:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13478.170440] LustreError: 445724:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [13478.218688] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13479.264439] Lustre: 443254:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775860025/real 1775860025] req@00000000dad4487e x1862123791754752/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1775860041 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13481.021389] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [13482.339513] LustreError: 469234:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13482.358028] LustreError: 469234:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13484.001118] Lustre: 443257:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775860030/real 1775860030] req@00000000c3c4aaf2 x1862123791754880/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1775860046 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13487.589217] LustreError: 469234:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13487.602942] LustreError: 469234:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13487.666597] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13487.935324] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13490.997257] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13491.001146] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [13495.460339] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13498.410943] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13513.702409] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13513.709739] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13513.721654] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13516.782243] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13516.787106] Lustre: Skipped 2 previous similar messages [13518.446362] Lustre: server umount lustre-OST0000 complete [13521.889099] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13521.892291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13521.901677] Lustre: Skipped 2 previous similar messages [13527.011338] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13527.019564] Lustre: Skipped 2 previous similar messages [13527.769214] Lustre: server umount lustre-MDT0000 complete [13531.020658] LustreError: 443977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775860093 with bad export cookie 5853816486335475550 [13531.028591] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13531.029892] LustreError: 443977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13531.324069] Lustre: server umount lustre-MDT0001 complete [13538.418904] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [13540.339988] Key type lgssc unregistered [13540.713871] LNet: 471771:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13540.723527] LNetError: 471771:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13540.731873] LNet: Removed LNI 192.168.203.137@tcp [13540.733640] LNet: Removed LNI 192.168.203.137@tcp10 [13540.738271] LNet: Removed LNI 192.168.203.137@tcp11 [13540.744344] LNet: Removed LNI 192.168.203.137@tcp12 [13540.749167] LNet: Removed LNI 192.168.203.137@tcp13 [13540.751829] LNet: Removed LNI 192.168.203.137@tcp14 [13540.758199] LNet: Removed LNI 192.168.203.137@tcp15 [13540.760359] LNet: Removed LNI 192.168.203.137@tcp16 [13540.762613] LNet: Removed LNI 192.168.203.137@tcp17 [13540.766357] LNet: Removed LNI 192.168.203.137@tcp18 [13540.771442] LNet: Removed LNI 192.168.203.137@tcp19 [13540.777181] LNet: Removed LNI 192.168.203.137@tcp20 [13540.784800] LNet: Removed LNI 192.168.203.137@tcp21 [13540.786823] LNet: Removed LNI 192.168.203.137@tcp22 [13540.793110] LNet: Removed LNI 192.168.203.137@tcp23 [13540.796226] LNet: Removed LNI 192.168.203.137@tcp24 [13540.798509] LNet: Removed LNI 192.168.203.137@tcp25 [13540.801160] LNet: Removed LNI 192.168.203.137@tcp26 [13540.805544] LNet: Removed LNI 192.168.203.137@tcp27 [13540.816147] LNet: Removed LNI 192.168.203.137@tcp28 [13540.819766] LNet: Removed LNI 192.168.203.137@tcp29 [13540.826185] LNet: Removed LNI 192.168.203.137@tcp30 [13540.831602] LNet: Removed LNI 192.168.203.137@tcp31 [13540.834366] LNet: Removed LNI 192.168.203.137@tcp32 [13540.838180] LNet: Removed LNI 192.168.203.137@tcp33 [13540.840472] LNet: Removed LNI 192.168.203.137@tcp34 [13540.847123] LNet: Removed LNI 192.168.203.137@tcp35 [13540.852647] LNet: Removed LNI 192.168.203.137@tcp36 [13540.857101] LNet: Removed LNI 192.168.203.137@tcp37 [13540.860815] LNet: Removed LNI 192.168.203.137@tcp38 [13540.864152] LNet: Removed LNI 192.168.203.137@tcp39 [13540.871061] LNet: Removed LNI 192.168.203.137@tcp40 [13540.873728] LNet: Removed LNI 192.168.203.137@tcp41 [13540.876105] LNet: Removed LNI 192.168.203.137@tcp42 [13540.877983] LNet: Removed LNI 192.168.203.137@tcp43 [13540.880663] LNet: Removed LNI 192.168.203.137@tcp44 [13540.887078] LNet: Removed LNI 192.168.203.137@tcp45 [13540.889429] LNet: Removed LNI 192.168.203.137@tcp46 [13540.895115] LNet: Removed LNI 192.168.203.137@tcp47 [13540.900239] LNet: Removed LNI 192.168.203.137@tcp48 [13540.902309] LNet: Removed LNI 192.168.203.137@tcp49 [13540.916958] LNet: Removed LNI 192.168.203.137@tcp50 [13541.441357] Key type .llcrypt unregistered [13541.442760] Key type ._llcrypt unregistered [13555.346919] Key type ._llcrypt registered [13555.348713] Key type .llcrypt registered [13555.417208] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [13556.229328] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13556.249044] alg: No test for adler32 (adler32-zlib) [13557.231173] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [13557.375780] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [13559.011093] Key type lgssc registered [13559.866524] Lustre: Echo OBD driver; http://www.lustre.org/ [13565.652741] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13565.663928] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13567.033677] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13570.117299] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [13576.700386] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13576.989366] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13580.043946] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [13584.091455] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13586.859942] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13592.370759] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13592.690608] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13593.716662] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [13597.884560] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [13602.249393] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13609.592513] loop0: detected capacity change from 0 to 204800000 [13609.696950] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13609.706726] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13609.720236] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13614.374575] loop0: detected capacity change from 0 to 204800000 [13614.518963] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13614.533105] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13614.542293] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13618.896875] loop0: detected capacity change from 0 to 204800000 [13623.037432] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13623.110631] Lustre: fs15246-OST0000: new disk, initializing [13623.113859] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [13623.156326] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13627.473762] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [13634.756847] loop1: detected capacity change from 0 to 204800000 [13638.508048] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13638.585953] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [13638.624614] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [13638.689799] Lustre: fs15246-MDT0000: new disk, initializing [13638.801117] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13638.820769] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13638.890794] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [13639.906603] Lustre: 477352:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000a69d2df5 x1862124300179584/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1775860218 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [13641.727603] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [13648.994688] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13649.000267] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13649.012587] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13658.090928] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13658.101609] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [13662.156671] Lustre: server umount fs15246-MDT0000 complete [13663.205791] LustreError: 472989:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13663.217212] LustreError: 472989:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [13668.167345] Lustre: server umount fs15246-OST0000 complete [13676.002725] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13676.009247] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13676.023063] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13678.564403] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13678.565959] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13683.685503] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13683.692210] Lustre: Skipped 2 previous similar messages [13688.802122] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13688.805463] Lustre: Skipped 1 previous similar message [13690.336124] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13690.423543] Lustre: server umount lustre-OST0000 complete [13693.922505] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13693.929315] Lustre: Skipped 1 previous similar message [13698.549834] Lustre: server umount lustre-MDT0000 complete [13699.042419] LustreError: 472993:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13699.049301] LustreError: 472993:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13700.558333] LustreError: 472974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775860263 with bad export cookie 587493254393293289 [13700.561687] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13700.563938] LustreError: 472974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13700.704670] Lustre: server umount lustre-MDT0001 complete [13705.729772] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [13707.183479] Key type lgssc unregistered [13707.348477] LNet: 480414:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13707.353819] LNetError: 480414:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13707.367066] LNet: Removed LNI 192.168.203.137@tcp [13707.774126] Key type .llcrypt unregistered [13707.776516] Key type ._llcrypt unregistered [13718.599704] Key type ._llcrypt registered [13718.603052] Key type .llcrypt registered [13718.679766] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [13719.269322] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13719.285971] alg: No test for adler32 (adler32-zlib) [13720.199955] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [13720.343944] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [13721.952181] Key type lgssc registered [13722.448567] Lustre: Echo OBD driver; http://www.lustre.org/ [13739.712911] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_hostid [13744.080882] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [13762.142222] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13762.171267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13763.328157] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13763.353976] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13763.402945] Lustre: lustre-MDT0000: new disk, initializing [13763.466549] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13763.477076] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13765.423138] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [13770.670697] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13770.721134] Lustre: 484803:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13770.755761] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13770.758471] Lustre: Skipped 1 previous similar message [13770.799792] Lustre: lustre-MDT0001: new disk, initializing [13770.845828] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13770.868668] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13770.874279] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13772.697583] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [13776.726257] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13778.691066] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13782.146269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13782.277033] Lustre: lustre-OST0000: new disk, initializing [13782.279875] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13782.325925] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13784.884845] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [13788.613698] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13790.508118] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [13790.726205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13790.733947] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13790.784842] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13791.615343] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [13793.151681] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [13793.233473] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13795.811808] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13795.812561] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13795.815215] LustreError: Skipped 1 previous similar message [13795.815988] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13795.821034] Lustre: Skipped 1 previous similar message [13795.829101] Lustre: Skipped 1 previous similar message [13800.138867] Lustre: server umount lustre-OST0000 complete [13806.049553] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13806.051481] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13806.055394] Lustre: Skipped 1 previous similar message [13806.060838] Lustre: Skipped 1 previous similar message [13811.169326] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13811.172198] Lustre: Skipped 1 previous similar message [13816.288770] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13816.289100] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13816.296717] Lustre: Skipped 1 previous similar message [13816.358511] Lustre: server umount lustre-MDT0000 complete [13818.005262] LustreError: 484794:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775860380 with bad export cookie 14770483875858036561 [13818.006784] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13818.011237] LustreError: 484794:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13818.182766] Lustre: server umount lustre-MDT0001 complete [13828.814204] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_hostid [13833.105659] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [13846.755921] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13677 sec ========= 18:33:27 (1775860407) [13847.553625] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 18:33:28 (1775860408) === [13849.114829] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 18:33:29 (1775860409) === [13866.359787] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [13867.591789] Key type lgssc unregistered [13867.752641] LNet: 492988:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13867.756532] LNetError: 492988:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13868.774850] LNet: Removed LNI 192.168.203.137@tcp [13869.155984] Key type .llcrypt unregistered [13869.158199] Key type ._llcrypt unregistered