[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 467803915 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.003260] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005017] kvm-guest: setup PV IPIs [ 0.008457] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009019] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.011023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.012013] pid_max: default: 32768 minimum: 301 [ 0.013113] LSM: Security Framework initializing [ 0.014049] Yama: becoming mindful. [ 0.015055] SELinux: Initializing. [ 0.016000] *** VALIDATE selinux *** [ 0.023114] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028317] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029180] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030119] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031177] *** VALIDATE tmpfs *** [ 0.033084] *** VALIDATE proc *** [ 0.034260] *** VALIDATE cgroup *** [ 0.035014] *** VALIDATE cgroup2 *** [ 0.036316] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038183] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039020] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040032] Spectre V2 : User space: Vulnerable [ 0.041012] Speculative Store Bypass: Vulnerable [ 0.044293] debug: unmapping init [mem 0xffffffff8da59000-0xffffffff8da60fff] [ 0.047219] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048785] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049029] ... version: 2 [ 0.050013] ... bit width: 48 [ 0.051013] ... generic registers: 4 [ 0.052014] ... value mask: 0000ffffffffffff [ 0.053012] ... max period: 00007fffffffffff [ 0.054010] ... fixed-purpose events: 3 [ 0.055010] ... event mask: 000000070000000f [ 0.056345] rcu: Hierarchical SRCU implementation. [ 0.058621] smp: Bringing up secondary CPUs ... [ 0.059558] x86: Booting SMP configuration: [ 0.060021] .... node #0, CPUs: #1 #2 #3 [ 0.063621] smp: Brought up 1 node, 4 CPUs [ 0.065013] smpboot: Max logical packages: 1 [ 0.066012] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.148296] node 0 deferred pages initialised in 79ms [ 0.150129] devtmpfs: initialized [ 0.151270] x86/mm: Memory block size: 128MB [ 0.153989] gcov: version magic: 0x41383552 [ 0.155314] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.156100] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.157492] pinctrl core: initialized pinctrl subsystem [ 0.158228] [ 0.158683] ************************************************************* [ 0.159016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.160018] ** ** [ 0.161021] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.162013] ** ** [ 0.163016] ** This means that this kernel is built to expose internal ** [ 0.164012] ** IOMMU data structures, which may compromise security on ** [ 0.165016] ** your system. ** [ 0.166015] ** ** [ 0.167009] ** If you see this message and you are not debugging the ** [ 0.168008] ** kernel, report this immediately to your vendor! ** [ 0.169010] ** ** [ 0.170008] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.171009] ************************************************************* [ 0.172634] NET: Registered protocol family 16 [ 0.173468] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.174108] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.175102] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.177179] cpuidle: using governor menu [ 0.180151] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.184029] PCI: Using configuration type 1 for base access [ 0.186188] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.197121] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.200022] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.203292] cryptd: max_cpu_qlen set to 1000 [ 0.206242] ACPI: Added _OSI(Module Device) [ 0.208016] ACPI: Added _OSI(Processor Device) [ 0.209021] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.211022] ACPI: Added _OSI(Processor Aggregator Device) [ 0.216737] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.226248] ACPI: Interpreter enabled [ 0.227067] ACPI: PM: (supports S0 S3 S4 S5) [ 0.228015] ACPI: Using IOAPIC for interrupt routing [ 0.229141] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.231727] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.244446] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.247047] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.250031] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.253091] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.258503] acpiphp: Slot [2] registered [ 0.260192] acpiphp: Slot [5] registered [ 0.262144] acpiphp: Slot [6] registered [ 0.263129] acpiphp: Slot [7] registered [ 0.265135] acpiphp: Slot [8] registered [ 0.266133] acpiphp: Slot [9] registered [ 0.268149] acpiphp: Slot [10] registered [ 0.270193] acpiphp: Slot [3] registered [ 0.272209] acpiphp: Slot [4] registered [ 0.273125] acpiphp: Slot [11] registered [ 0.275134] acpiphp: Slot [12] registered [ 0.277158] acpiphp: Slot [13] registered [ 0.279127] acpiphp: Slot [14] registered [ 0.280117] acpiphp: Slot [15] registered [ 0.282139] acpiphp: Slot [16] registered [ 0.284132] acpiphp: Slot [17] registered [ 0.285093] acpiphp: Slot [18] registered [ 0.286150] acpiphp: Slot [19] registered [ 0.288186] acpiphp: Slot [20] registered [ 0.290177] acpiphp: Slot [21] registered [ 0.292192] acpiphp: Slot [22] registered [ 0.294158] acpiphp: Slot [23] registered [ 0.296232] acpiphp: Slot [24] registered [ 0.298152] acpiphp: Slot [25] registered [ 0.299164] acpiphp: Slot [26] registered [ 0.301164] acpiphp: Slot [27] registered [ 0.303155] acpiphp: Slot [28] registered [ 0.305239] acpiphp: Slot [29] registered [ 0.307159] acpiphp: Slot [30] registered [ 0.309150] acpiphp: Slot [31] registered [ 0.310182] PCI host bridge to bus 0000:00 [ 0.312031] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.315034] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.317032] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.320037] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.323041] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.326081] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.328245] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.332215] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.335811] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.346884] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.353031] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.355033] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.358029] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.361040] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.364718] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.367920] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.372063] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.375598] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.382023] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.401028] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.407022] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.413525] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.422023] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.430496] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.450023] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.467479] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.479022] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.488024] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.511029] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.522284] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.531026] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.540024] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.563025] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.575979] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.585025] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.593025] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.610020] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.622332] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.633029] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.641022] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.656020] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.670076] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.674015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.679020] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.693016] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.701458] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.704370] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.706335] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.708320] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.710221] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.715047] iommu: Default domain type: Passthrough [ 0.717561] SCSI subsystem initialized [ 0.719202] ACPI: bus type USB registered [ 0.721154] usbcore: registered new interface driver usbfs [ 0.723114] usbcore: registered new interface driver hub [ 0.725114] usbcore: registered new device driver usb [ 0.727255] pps_core: LinuxPPS API ver. 1 registered [ 0.729012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.733090] PTP clock support registered [ 0.736058] EDAC MC: Ver: 3.0.0 [ 0.738165] PCI: Using ACPI for IRQ routing [ 0.739991] NetLabel: Initializing [ 0.741021] NetLabel: domain hash size = 128 [ 0.743016] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.744099] NetLabel: unlabeled traffic allowed by default [ 0.746156] vgaarb: loaded [ 0.748276] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.749011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.755000] clocksource: Switched to clocksource kvm-clock [ 0.866576] VFS: Disk quotas dquot_6.6.0 [ 0.868343] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.871208] *** VALIDATE ramfs *** [ 0.872505] *** VALIDATE hugetlbfs *** [ 0.874266] pnp: PnP ACPI init [ 0.877069] pnp: PnP ACPI: found 6 devices [ 0.894455] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.898240] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.900080] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.901883] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.904154] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.906411] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.909637] NET: Registered protocol family 2 [ 0.912399] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.917532] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.921402] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.951197] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.958779] TCP: Hash tables configured (established 65536 bind 65536) [ 0.961869] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.964543] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.967033] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.969656] NET: Registered protocol family 1 [ 0.974690] RPC: Registered named UNIX socket transport module. [ 0.976434] RPC: Registered udp transport module. [ 0.977970] RPC: Registered tcp transport module. [ 0.979472] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.981381] NET: Registered protocol family 44 [ 0.982982] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.985414] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.987674] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.990205] PCI: CLS 0 bytes, default 64 [ 0.991989] Unpacking initramfs... [ 2.413707] debug: unmapping init [mem 0xffff8d0a3cc54000-0xffff8d0a3ffbffff] [ 2.417823] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.419825] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.423312] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.920126] Initialise system trusted keyrings [ 2.922729] Key type blacklist registered [ 2.925132] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.936959] zbud: loaded [ 2.941273] *** VALIDATE nfs *** [ 2.942806] *** VALIDATE nfs4 *** [ 2.944506] pstore: using deflate compression [ 2.948217] Platform Keyring initialized [ 3.062962] NET: Registered protocol family 38 [ 3.064864] Key type asymmetric registered [ 3.066342] Asymmetric key parser 'x509' registered [ 3.068203] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.071225] io scheduler mq-deadline registered [ 3.073127] io scheduler kyber registered [ 3.075182] io scheduler bfq registered [ 3.077624] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.080888] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.084417] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.087917] ACPI: Power Button [PWRF] [ 3.093760] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.101348] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.119245] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.127962] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.151206] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.181997] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.210797] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.216141] Non-volatile memory driver v1.3 [ 3.217747] Linux agpgart interface v0.103 [ 3.251858] virtio_blk virtio1: [vda] 149960 512-byte logical blocks (76.8 MB/73.2 MiB) [ 3.256462] vda: detected capacity change from 0 to 76779520 [ 3.272294] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.275619] vdb: detected capacity change from 0 to 1073741824 [ 3.296339] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.299346] vdc: detected capacity change from 0 to 2621440000 [ 3.315328] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.318968] vdd: detected capacity change from 0 to 2621440000 [ 3.333257] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.336245] vde: detected capacity change from 0 to 4294967296 [ 3.358087] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.361335] vdf: detected capacity change from 0 to 4294967296 [ 3.369106] libphy: Fixed MDIO Bus: probed [ 3.374646] usbcore: registered new interface driver usbserial_generic [ 3.377233] usbserial: USB Serial support registered for generic [ 3.379972] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.385545] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.387780] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.390219] mousedev: PS/2 mouse device common for all mice [ 3.393108] rtc_cmos 00:05: RTC can wake from S4 [ 3.396431] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.396503] rtc_cmos 00:05: registered as rtc0 [ 3.401149] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.405021] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.407258] intel_pstate: CPU model not supported [ 3.409784] hid: raw HID events driver (C) Jiri Kosina [ 3.415079] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.416477] usbcore: registered new interface driver usbhid [ 3.420800] usbhid: USB HID core driver [ 3.422516] drop_monitor: Initializing network drop monitor service [ 3.425065] Initializing XFRM netlink socket [ 3.427613] NET: Registered protocol family 10 [ 3.430744] Segment Routing with IPv6 [ 3.432864] NET: Registered protocol family 17 [ 3.435184] mpls_gso: MPLS GSO support [ 3.441893] RAS: Correctable Errors collector initialized. [ 3.444603] AVX version of gcm_enc/dec engaged. [ 3.446717] AES CTR mode by8 optimization enabled [ 3.523920] sched_clock: Marking stable (3523891451, 0)->(4461186491, -937295040) [ 3.527832] registered taskstats version 1 [ 3.530154] Loading compiled-in X.509 certificates [ 3.532628] zswap: loaded using pool lzo/zbud [ 3.559599] Key type big_key registered [ 3.572163] Key type encrypted registered [ 3.573596] ima: No TPM chip found, activating TPM-bypass! [ 3.575332] ima: Allocated hash algorithm: sha1 [ 3.577296] ima: No architecture policies found [ 3.579382] evm: Initialising EVM extended attributes: [ 3.581775] evm: security.selinux [ 3.583267] evm: security.ima [ 3.584774] evm: security.capability [ 3.586269] evm: HMAC attrs: 0x1 [ 3.589298] rtc_cmos 00:05: setting system clock to 2026-09-16 00:19:23 UTC (1789517963) [ 3.595893] debug: unmapping init [mem 0xffffffff8ea03000-0xffffffff8ebfffff] [ 3.599647] debug: unmapping init [mem 0xffffffff8d782000-0xffffffff8da58fff] [ 3.609138] Write protecting the kernel read-only data: 28672k [ 3.612604] debug: unmapping init [mem 0xffffffff8be03000-0xffffffff8bffffff] [ 3.615657] debug: unmapping init [mem 0xffffffff8c714000-0xffffffff8c7fffff] [ 3.648291] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.657239] systemd[1]: Detected virtualization kvm. [ 3.659455] systemd[1]: Detected architecture x86-64. [ 3.661281] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.689332] systemd[1]: No hostname configured. [ 3.691037] systemd[1]: Set hostname to . [ 3.692861] random: systemd: uninitialized urandom read (16 bytes read) [ 3.695490] systemd[1]: Initializing machine ID from random generator. [ 3.757234] random: ln: uninitialized urandom read (6 bytes read) [ 3.837070] random: systemd: uninitialized urandom read (16 bytes read) [ 3.842299] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.848297] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.852790] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Slices. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Swap. [ OK ] Reached target Sockets. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.549248] device-mapper: uevent: version 1.0.3 [ 4.552128] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.253273] virtio_net virtio0 ens2: renamed from eth0 [ 5.271448] random: fast init done [ 5.460281] scsi host0: ata_piix [ 5.513112] scsi host1: ata_piix [ 5.514846] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.517614] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.738597] dracut-initqueue[587]: RTNETLINK answers: File exists [ 10.091380] random: crng init done [ 10.092914] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.524963] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.690313] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.977079] SELinux: Disabled at runtime. [ 12.040565] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.050887] systemd[1]: Detected virtualization kvm. [ 12.053270] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.589595] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.593401] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.600170] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.604283] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.607655] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.616303] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.626568] systemd[1]: Mounting Huge Pages File System... Mounting Huge Pages File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Control Socket. Mounting POSIX Message Queue File System... [ OK ] Stopped target Switch Root. Mounting Kernel Debug File System... Starting Apply Kernel Variables... [ OK ] Created slice system-getty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-sshd\x2dkeygen.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice User and Session Slice. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target Slices. [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Initrd Root File System. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 12.752989] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 13.002589] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 13.366769] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.386920] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.497269] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.506226] EDAC sbridge: Ver: 1.1.2 [ 17.257401] Key type dns_resolver registered [ 17.964737] NFS: Registering the id_resolver key type [ 17.969965] Key type id_resolver registered [ 17.971638] Key type id_legacy registered [* ] A start job is running for Configur…-only root support (5s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Load/Save Random Seed. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg407-server login: [ 79.997354] hrtimer: interrupt took 2344182 ns [ 81.958885] libcfs: loading out-of-tree module taints kernel. [ 82.037418] Key type ._llcrypt registered [ 82.042158] Key type .llcrypt registered [ 82.165713] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [ 105.189522] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 106.763885] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 106.776812] alg: No test for adler32 (adler32-zlib) [ 108.100221] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 108.838142] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 110.583246] Key type lgssc registered [ 112.880230] Lustre: Echo OBD driver; http://www.lustre.org/ [ 135.150524] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 182.257980] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 198.904645] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 198.988850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 200.340375] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 200.404458] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 200.549502] Lustre: lustre-MDT0000: new disk, initializing [ 200.705024] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 200.741098] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 205.300884] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 219.048551] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 219.153536] Lustre: 6512:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 219.188473] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 219.191456] Lustre: Skipped 1 previous similar message [ 219.269601] Lustre: lustre-MDT0001: new disk, initializing [ 219.331338] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 219.362159] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 219.373948] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 223.687338] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 228.526671] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 238.434053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 238.747055] Lustre: lustre-OST0000: new disk, initializing [ 238.756774] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 238.767226] Lustre: 8448:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 238.853177] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 244.788168] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 244.804694] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 244.914604] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 245.535854] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 259.719924] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 259.850488] Lustre: lustre-OST0001: new disk, initializing [ 259.858086] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 259.866225] Lustre: 9522:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 259.927455] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 266.297154] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 266.307550] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 266.309564] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 266.455460] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 277.025802] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 284.369957] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 290.849456] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing check_logdir /tmp/testlogs/ [ 295.862381] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing yml_node [ 300.652633] Lustre: DEBUG MARKER: Client: 2.17.58.39 [ 303.571088] Lustre: DEBUG MARKER: MDS: 2.17.58.39 [ 306.389677] Lustre: DEBUG MARKER: OSS: 2.17.58.39 [ 308.513568] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Sep 15 20:24:25 EDT 2026 [ 327.289783] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 328.668693] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 347.621245] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 347.629198] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 347.640782] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 348.645561] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 348.660098] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 348.672120] Lustre: Skipped 3 previous similar messages [ 353.764902] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 353.770437] Lustre: Skipped 3 previous similar messages [ 358.368779] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 358.602036] Lustre: server umount lustre-MDT0000 complete [ 362.014163] LustreError: 6503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789518321 with bad export cookie 6770622771732662378 [ 362.018420] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 362.030187] LustreError: 6503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 364.000957] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 364.001778] LustreError: 6516:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 364.019337] Lustre: Skipped 2 previous similar messages [ 364.051864] LustreError: 6516:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 364.063527] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 368.032332] LustreError: 11182:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 368.046050] LustreError: 11182:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 368.349709] Lustre: server umount lustre-MDT0001 complete [ 372.307911] Lustre: server umount lustre-OST0000 complete [ 376.174534] Lustre: server umount lustre-OST0001 complete [ 383.906486] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [ 394.219738] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 439.785399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 440.047428] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 440.096691] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 440.211523] Lustre: lustre-MDT0000: new disk, initializing [ 440.297335] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 440.322634] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 445.440888] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 456.417690] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 456.602284] Lustre: 15074:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 456.648550] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 456.652107] Lustre: Skipped 1 previous similar message [ 456.743780] Lustre: lustre-MDT0001: new disk, initializing [ 456.893537] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 456.952593] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 456.968115] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 461.950421] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 473.494121] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 480.944373] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 488.156152] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 488.357673] Lustre: lustre-OST0000: new disk, initializing [ 488.361947] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 488.372263] Lustre: 17042:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 488.484795] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 489.579686] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 489.590902] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 489.785490] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 494.083021] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 504.193300] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 508.538945] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 508.749158] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 513.281597] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 513.451365] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 515.562244] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 515.562305] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 515.569974] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 515.569982] Lustre: Skipped 3 previous similar messages [ 515.574617] LustreError: Skipped 1 previous similar message [ 515.609047] Lustre: Skipped 2 previous similar messages [ 525.795916] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 525.804930] Lustre: Skipped 4 previous similar messages [ 529.375163] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 529.622043] Lustre: server umount lustre-OST0000 complete [ 533.984887] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 534.001863] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 546.274203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 546.277770] Lustre: Skipped 6 previous similar messages [ 548.319522] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 548.663376] Lustre: server umount lustre-MDT0000 complete [ 551.402779] LustreError: 15087:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 551.430578] LustreError: 15087:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 553.131270] LustreError: 15069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789518513 with bad export cookie 6770622771732663911 [ 553.139985] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 553.141846] LustreError: 15069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 553.443252] Lustre: server umount lustre-MDT0001 complete [ 558.188744] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 20:28:35 (1789518515) [ 560.694543] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 562.395230] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 20:28:39 (1789518519) [ 568.465628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 568.959483] LustreError: 19455:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 569.063762] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 573.089789] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 574.438179] LustreError: 19456:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 574.465166] LustreError: 19456:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 581.363715] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 585.894872] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 593.324965] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 599.307781] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 606.146882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 612.663565] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 619.200989] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 628.193086] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 628.197175] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 628.203334] Lustre: Skipped 1 previous similar message [ 628.206901] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 631.464680] Lustre: server umount lustre-OST0000 complete [ 637.922359] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 637.938541] LustreError: Skipped 1 previous similar message [ 641.292110] Lustre: server umount lustre-MDT0000 complete [ 644.576965] LustreError: 19456:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 644.602171] LustreError: 19456:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 645.207704] LustreError: 19435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789518605 with bad export cookie 6770622771732664793 [ 645.209867] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 645.224930] LustreError: 19435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 645.595633] Lustre: server umount lustre-MDT0001 complete [ 656.732643] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 659.556310] Key type lgssc unregistered [ 659.888760] LNet: 22986:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 659.897115] LNetError: 22986:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 659.926503] LNet: Removed LNI 192.168.204.107@tcp [ 660.983315] Key type .llcrypt unregistered [ 660.986537] Key type ._llcrypt unregistered [ 684.480570] Key type ._llcrypt registered [ 684.484831] Key type .llcrypt registered [ 684.603329] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 685.566278] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 685.651400] alg: No test for adler32 (adler32-zlib) [ 686.757993] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 687.030570] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 688.776217] Key type lgssc registered [ 690.172904] Lustre: Echo OBD driver; http://www.lustre.org/ [ 698.622314] loop: module loaded [ 698.698363] loop0: detected capacity change from 0 to 204800000 [ 698.959340] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 698.969336] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 699.255377] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 705.270679] loop0: detected capacity change from 0 to 204800000 [ 705.451970] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 705.461111] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 705.470220] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 710.319944] loop0: detected capacity change from 0 to 204800000 [ 715.590519] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 715.612555] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 716.827763] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 716.865836] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 716.922728] Lustre: lustre-MDT0000: new disk, initializing [ 717.030655] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 717.059311] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 721.358153] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 730.355733] loop1: detected capacity change from 0 to 204800000 [ 736.742995] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 737.101125] Lustre: lustre-OST07c6: new disk, initializing [ 737.107523] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 737.115204] Lustre: Skipped 1 previous similar message [ 737.121778] Lustre: 26170:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 737.246331] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 744.534895] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 744.541907] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 744.565486] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 744.655234] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 750.403599] Lustre: 26381:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 750.412799] Lustre: 26381:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 750.419223] Lustre: 26381:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 750.425958] Lustre: 26381:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 750.430919] Lustre: 26381:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 750.435771] Lustre: 26381:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 763.674802] Lustre: 25157:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 763.687222] Lustre: 25157:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 763.695823] Lustre: 25157:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 763.710070] Lustre: 25157:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 763.724293] Lustre: 25157:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 763.734074] Lustre: 25157:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 768.543902] Lustre: 25157:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 768.555424] Lustre: 25157:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 768.563299] Lustre: 25157:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 768.571674] Lustre: 25157:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 768.580183] Lustre: 25157:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 768.588346] Lustre: 25157:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 768.595212] Lustre: 25157:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 768.608625] Lustre: 25157:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 768.621193] Lustre: 25157:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 768.631827] Lustre: 25157:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 768.635441] Lustre: 25157:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 768.640643] Lustre: 25157:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 771.828450] Lustre: 26381:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 771.853110] Lustre: 26381:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 771.863217] Lustre: 26381:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 771.875284] Lustre: 26381:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 771.880750] Lustre: 26381:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 771.887308] Lustre: 26381:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 771.901519] Lustre: 26381:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 771.907090] Lustre: 26381:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 771.914045] Lustre: 26381:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 771.923567] Lustre: 26381:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 771.929283] Lustre: 26381:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 771.941457] Lustre: 26381:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 776.952310] Lustre: 25158:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 776.963887] Lustre: 25158:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 776.968519] Lustre: 25158:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 776.972870] Lustre: 25158:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 776.977351] Lustre: 25158:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 776.982764] Lustre: 25158:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 776.990649] Lustre: 25158:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 776.995611] Lustre: 25158:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 777.000708] Lustre: 25158:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 777.006328] Lustre: 25158:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 777.010607] Lustre: 25158:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 777.015685] Lustre: 25158:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 785.123370] Lustre: 25156:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 257 < left 278, rollback = 2 [ 785.141258] Lustre: 25156:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages [ 785.146872] Lustre: 25156:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 785.154363] Lustre: 25156:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 785.163659] Lustre: 25156:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 785.169306] Lustre: 25156:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 785.176806] Lustre: 25156:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 785.190267] Lustre: 25156:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 785.203840] Lustre: 25156:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 785.213282] Lustre: 25156:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 785.223988] Lustre: 25156:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 785.252515] Lustre: 25156:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 841.711613] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 841.714944] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 841.737368] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 843.118131] Lustre: server umount lustre-OST07c6 complete [ 853.253996] Lustre: server umount lustre-MDT0000 complete [ 870.901417] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 873.908536] Key type lgssc unregistered [ 874.174637] LNet: 28723:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 874.186316] LNetError: 28723:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 874.207971] LNet: Removed LNI 192.168.204.107@tcp [ 875.029135] Key type .llcrypt unregistered [ 875.032519] Key type ._llcrypt unregistered [ 894.842518] Key type ._llcrypt registered [ 894.844231] Key type .llcrypt registered [ 895.008614] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 896.302871] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 896.580378] alg: No test for adler32 (adler32-zlib) [ 897.655644] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 897.955023] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 899.615392] Key type lgssc registered [ 901.512561] Lustre: Echo OBD driver; http://www.lustre.org/ [ 912.529638] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 912.553417] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 914.212147] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 922.032682] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 924.642068] LustreError: 29994:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 924.671299] LustreError: 29994:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 929.766846] LustreError: 29995:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 932.151129] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 932.654349] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 938.131778] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 947.929732] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 954.745901] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 964.272180] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 964.808492] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 972.705253] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 980.699462] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 984.327299] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 1004.002479] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1004.017134] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1004.048802] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1005.028847] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1005.031221] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1005.073155] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1006.635862] Lustre: server umount lustre-OST0000 complete [ 1014.757634] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1014.770298] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1014.787951] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1014.800080] Lustre: Skipped 1 previous similar message [ 1017.132762] Lustre: server umount lustre-MDT0000 complete [ 1020.384942] LustreError: 29989:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1020.444797] LustreError: 29989:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1021.783160] LustreError: 29975:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789518981 with bad export cookie 8000394462460006972 [ 1021.798738] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1021.818425] LustreError: 29975:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1022.375675] Lustre: server umount lustre-MDT0001 complete [ 1034.738236] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 1037.675904] Key type lgssc unregistered [ 1037.908334] LNet: 33733:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1037.912894] LNetError: 33733:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1037.921956] LNet: Removed LNI 192.168.204.107@tcp [ 1038.818367] Key type .llcrypt unregistered [ 1038.827070] Key type ._llcrypt unregistered [ 1058.643672] Key type ._llcrypt registered [ 1058.646880] Key type .llcrypt registered [ 1058.776867] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 1059.674770] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1059.709267] alg: No test for adler32 (adler32-zlib) [ 1060.869146] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 1061.188649] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 1062.903296] Key type lgssc registered [ 1064.096699] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1071.456174] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1071.475539] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1072.801710] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1076.477485] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1082.343686] LustreError: 34999:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1082.373835] LustreError: 34999:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1084.806833] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1085.162194] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1089.534555] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1096.322609] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1101.526303] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1109.475504] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1110.043801] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1117.245227] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1118.060356] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1128.698904] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1140.709156] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1140.716847] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1140.722860] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1142.753870] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1142.769284] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1142.776225] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1142.808479] Lustre: Skipped 1 previous similar message [ 1146.286293] Lustre: server umount lustre-OST0000 complete [ 1151.973386] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1151.985678] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1152.002932] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1158.114454] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1158.120984] Lustre: Skipped 3 previous similar messages [ 1163.233390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1164.255648] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1164.537634] Lustre: server umount lustre-MDT0000 complete [ 1167.336595] LustreError: 34998:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1167.361044] LustreError: 34998:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1168.618040] LustreError: 34980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789519128 with bad export cookie 87041919099005493 [ 1168.634731] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1169.013575] Lustre: server umount lustre-MDT0001 complete [ 1180.735692] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 1183.807905] Key type lgssc unregistered [ 1184.227041] LNet: 38531:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1184.239524] LNetError: 38531:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1184.259925] LNet: Removed LNI 192.168.204.107@tcp [ 1185.420656] Key type .llcrypt unregistered [ 1185.422960] Key type ._llcrypt unregistered [ 1211.396631] Key type ._llcrypt registered [ 1211.399833] Key type .llcrypt registered [ 1211.521803] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 1212.502474] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1212.616377] alg: No test for adler32 (adler32-zlib) [ 1213.742972] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 1213.977553] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 1215.647311] Key type lgssc registered [ 1216.560714] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1224.273263] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1224.287463] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1225.739124] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1230.313571] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1235.937858] LustreError: 39945:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1235.968113] LustreError: 39945:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1238.281653] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1238.654150] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1242.943236] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1250.959564] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1256.973764] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1264.840094] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1265.215522] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1270.313843] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1271.467522] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1279.978458] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1292.901209] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.7@tcp (stopping) [ 1294.315418] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1294.318405] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1294.341264] Lustre: Skipped 2 previous similar messages [ 1294.354403] Lustre: Skipped 2 previous similar messages [ 1296.377813] Lustre: server umount lustre-MDT0000 complete [ 1297.981319] LustreError: 39942:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1299.425273] LustreError: 39945:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1299.448536] LustreError: 39945:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1300.679203] LustreError: 39925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789519260 with bad export cookie 14423465783785555750 [ 1300.688630] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1300.708573] LustreError: 39925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1303.102833] LustreError: 41787:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1303.165068] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.7@tcp (stopping) [ 1304.544440] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1305.570639] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1305.587614] Lustre: Skipped 1 previous similar message [ 1307.377707] Lustre: server umount lustre-MDT0001 complete [ 1322.960173] Lustre: server umount lustre-OST0000 complete [ 1341.323097] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 1344.281329] Key type lgssc unregistered [ 1344.655891] LNet: 43808:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1344.663925] LNetError: 43808:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1344.678533] LNet: Removed LNI 192.168.204.107@tcp [ 1345.663156] Key type .llcrypt unregistered [ 1345.664985] Key type ._llcrypt unregistered [ 1372.420860] Key type ._llcrypt registered [ 1372.426697] Key type .llcrypt registered [ 1372.579352] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 1374.004792] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1374.021956] alg: No test for adler32 (adler32-zlib) [ 1375.102662] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 1375.406382] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 1377.113260] Key type lgssc registered [ 1378.371333] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1386.056186] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1386.092638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1387.658594] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1392.917601] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1397.225293] LustreError: 45222:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1397.242431] LustreError: 45222:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1402.344487] LustreError: 45223:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1403.105702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1403.555898] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1409.827668] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1419.926996] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1426.732708] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1434.050282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1434.510905] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1442.848339] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1450.862777] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1459.134491] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1469.415365] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1469.429895] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1469.438126] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1469.455611] Lustre: Skipped 1 previous similar message [ 1472.125927] Lustre: lustre-OST0000: Not available for connect from 192.168.204.7@tcp (stopping) [ 1472.134610] Lustre: Skipped 1 previous similar message [ 1473.260979] Lustre: server umount lustre-OST0000 complete [ 1495.018206] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1495.026621] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1495.039903] Lustre: Skipped 1 previous similar message [ 1495.064332] Lustre: Skipped 1 previous similar message [ 1500.129155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1500.134399] Lustre: Skipped 1 previous similar message [ 1505.249789] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1505.264619] Lustre: Skipped 1 previous similar message [ 1507.295354] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1507.485505] Lustre: server umount lustre-MDT0000 complete [ 1509.347061] LustreError: 45222:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1509.370248] LustreError: 45222:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1511.739729] LustreError: 47066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789519471 with bad export cookie 17467011795220284393 [ 1511.744811] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1511.753943] LustreError: 47066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1526.240393] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1526.534662] Lustre: server umount lustre-MDT0001 complete [ 1537.648694] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 1540.450723] Key type lgssc unregistered [ 1540.804577] LNet: 48962:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1540.815069] LNetError: 48962:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1540.831556] LNet: Removed LNI 192.168.204.107@tcp [ 1541.580337] Key type .llcrypt unregistered [ 1541.584336] Key type ._llcrypt unregistered [ 1567.644072] Key type ._llcrypt registered [ 1567.647156] Key type .llcrypt registered [ 1567.868897] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 1569.201948] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1569.334702] alg: No test for adler32 (adler32-zlib) [ 1570.597913] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 1570.917469] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 1572.607274] Key type lgssc registered [ 1574.163336] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1584.015930] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1584.063876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1585.771520] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1591.900300] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1595.877979] LustreError: 50376:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1595.905180] LustreError: 50376:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1600.996345] LustreError: 50377:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1602.729239] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1603.127775] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1608.830706] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1616.776749] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1623.110527] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1630.908401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1631.372785] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1637.579259] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1645.423305] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1655.991800] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1661.429645] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1665.041721] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1677.099662] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-1520h47m14s [ 1678.675259] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1681.892740] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1681.903622] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1681.906688] Lustre: Skipped 1 previous similar message [ 1681.922038] Lustre: Skipped 2 previous similar messages [ 1682.978695] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1682.987391] Lustre: Skipped 2 previous similar messages [ 1684.094752] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.7@tcp (stopping) [ 1684.103944] Lustre: Skipped 1 previous similar message [ 1687.478269] Lustre: server umount lustre-MDT0000 complete [ 1689.083483] LustreError: 50371:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1689.105731] LustreError: 50371:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1691.567438] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1695.209891] LustreError: 50376:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1695.220745] LustreError: 50376:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 1697.541269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1697.630898] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1697.893946] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1697.961189] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1698.925571] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1702.061911] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1705.086093] Lustre: DEBUG MARKER: Wait for df (36675) ... [ 1706.574079] Lustre: DEBUG MARKER: done [ 1713.126184] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1713.137432] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1713.145815] Lustre: Skipped 1 previous similar message [ 1713.153091] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1714.669821] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1714.670244] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1719.778958] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1719.793075] Lustre: Skipped 3 previous similar messages [ 1724.384100] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1724.561398] Lustre: server umount lustre-OST0000 complete [ 1730.017582] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1730.026938] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1730.037864] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1734.881949] Lustre: server umount lustre-MDT0000 complete [ 1737.185053] LustreError: 52866:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1737.222489] LustreError: 52866:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1739.357372] LustreError: 50357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789519699 with bad export cookie 14152815441251291834 [ 1739.381103] LustreError: 50357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1739.384283] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1739.825995] Lustre: server umount lustre-MDT0001 complete [ 1753.816467] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 1756.918982] Key type lgssc unregistered [ 1757.315042] LNet: 55322:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1757.324169] LNetError: 55322:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1757.374539] LNet: Removed LNI 192.168.204.107@tcp [ 1758.494862] Key type .llcrypt unregistered [ 1758.497436] Key type ._llcrypt unregistered [ 1804.609415] Key type ._llcrypt registered [ 1804.611483] Key type .llcrypt registered [ 1804.713404] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 1805.733587] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1805.799478] alg: No test for adler32 (adler32-zlib) [ 1806.795489] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 1807.039949] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 1808.735785] Key type lgssc registered [ 1810.087620] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1819.205578] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1819.237564] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1820.615599] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1820.638290] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1820.809733] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1826.709776] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1837.446112] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1837.581478] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1837.619369] Lustre: 57528:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1837.947147] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1842.695299] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1851.803893] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1858.749458] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1866.704369] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1866.917724] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1867.216451] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1874.545905] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1875.679039] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1882.400299] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1889.705501] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-1520h50m47s [ 1891.430681] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1899.625322] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1902.378545] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1902.427859] LustreError: 59801:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1922.674248] Lustre: lustre-MDT0000: Client 681cb1ac-7e60-48c3-97c4-922859530a1a (at 192.168.204.7@tcp) reconnecting [ 1942.447294] LustreError: 59801:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1944.934098] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1947.669400] Lustre: DEBUG MARKER: done [ 1957.349707] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1957.358442] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1957.372423] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1957.872742] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1957.879367] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1960.384226] Lustre: server umount lustre-OST0000 complete [ 1966.052567] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1966.061702] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1966.082699] Lustre: Skipped 1 previous similar message [ 1966.094832] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1970.625667] Lustre: server umount lustre-MDT0000 complete [ 1975.088286] LustreError: 57521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789519934 with bad export cookie 10815817476410329825 [ 1975.092991] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1975.112076] LustreError: 57521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1975.641842] Lustre: server umount lustre-MDT0001 complete [ 1988.788194] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 1992.984872] Key type lgssc unregistered [ 1993.484862] LNet: 61860:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1993.512984] LNetError: 61860:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1993.544039] LNet: Removed LNI 192.168.204.107@tcp [ 1994.560449] Key type .llcrypt unregistered [ 1994.573071] Key type ._llcrypt unregistered [ 2047.215567] Key type ._llcrypt registered [ 2047.218371] Key type .llcrypt registered [ 2047.326573] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 2048.421086] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2048.503045] alg: No test for adler32 (adler32-zlib) [ 2049.658531] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 2050.034590] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 2051.738799] Key type lgssc registered [ 2053.131927] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2056.673780] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2056.697903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2058.060956] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2058.087914] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2058.282242] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2059.545601] Lustre: Failing over lustre-MDT0000 [ 2059.949745] Lustre: server umount lustre-MDT0000 complete [ 2066.321244] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 20:53:43 (1789520023) [ 2072.964021] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2073.316967] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2077.565954] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2085.890062] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2086.008739] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2086.060738] Lustre: 64658:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2086.253547] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2090.704106] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2098.077251] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2104.088316] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2111.296654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2111.512214] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2111.736657] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2118.045580] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2121.404473] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2126.927959] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2133.227749] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2142.183754] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2142.193753] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2142.220304] Lustre: Skipped 2 previous similar messages [ 2142.228809] Lustre: Skipped 2 previous similar messages [ 2145.625524] Lustre: server umount lustre-MDT0000 complete [ 2149.706473] LustreError: 66514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789520109 with bad export cookie 4931088520762753976 [ 2149.711655] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2149.718094] LustreError: 66514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2150.190472] Lustre: server umount lustre-MDT0001 complete [ 2154.022470] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2158.924764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2167.564349] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2168.799713] Lustre: 63579:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789520112/real 1789520112] req@00000000dda43710 x1876447732998656/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789520128 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2168.834895] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2170.851606] Lustre: 63581:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789520114/real 1789520114] req@0000000032dee2f6 x1876447732998784/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789520130 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2173.983892] Lustre: 63581:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789520117/real 1789520117] req@0000000057df7328 x1876447732998912/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789520133 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2175.320410] LustreError: 66508:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2175.333058] LustreError: 66508:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2175.408830] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2175.509635] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2176.418593] Lustre: 63578:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789520119/real 1789520119] req@0000000032afedd2 x1876447732999168/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789520135 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2177.444690] LustreError: 66509:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2178.474403] LustreError: 68427:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2178.502910] LustreError: 68427:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2180.416676] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2180.511985] Lustre: 63579:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789520124/real 1789520124] req@000000007170fbb8 x1876447732999296/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789520140 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2183.660876] LustreError: 66508:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2183.693825] LustreError: 66508:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2188.775970] LustreError: 66509:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2188.797437] LustreError: 66509:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2189.374617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2189.749081] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2194.537408] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2194.925611] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2197.996341] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2201.730379] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2207.311297] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2218.467118] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2218.470083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2218.477796] Lustre: Skipped 2 previous similar messages [ 2218.487295] Lustre: Skipped 1 previous similar message [ 2221.863890] Lustre: server umount lustre-MDT0000 complete [ 2223.586201] LustreError: 68613:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2223.602253] LustreError: 68613:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2226.116559] LustreError: 64651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789520186 with bad export cookie 4931088520762756251 [ 2226.122272] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2226.132618] LustreError: 64651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2226.796701] Lustre: server umount lustre-MDT0001 complete [ 2234.927732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2245.087638] Lustre: 63579:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789520188/real 1789520188] req@0000000040a3e87c x1876447733027712/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789520204 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2245.137992] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2245.400323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2251.702171] LustreError: 66509:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2251.805394] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2251.872555] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2257.580867] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2267.183944] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2272.428254] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2273.262161] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2280.423806] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2286.004627] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2292.273564] Lustre: server umount lustre-MDT0000 complete [ 2293.216675] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2293.229209] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2293.245484] LustreError: 69162:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2293.272235] LustreError: 69162:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 2295.682375] LustreError: 64649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789520255 with bad export cookie 4931088520762757630 [ 2295.684858] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2295.694847] LustreError: 64649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2298.096470] Lustre: server umount lustre-MDT0001 complete [ 2304.918654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2308.556127] Lustre: DEBUG MARKER: files compared the same [ 2321.385564] Lustre: server umount lustre-OST0000 complete [ 2338.245436] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 2341.698072] Key type lgssc unregistered [ 2342.263585] LNet: 74386:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2342.275686] LNetError: 74386:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2342.306796] LNet: Removed LNI 192.168.204.107@tcp [ 2343.642206] Key type .llcrypt unregistered [ 2343.646641] Key type ._llcrypt unregistered [ 2369.331682] Key type ._llcrypt registered [ 2369.334746] Key type .llcrypt registered [ 2369.420812] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 2371.081089] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2371.105478] alg: No test for adler32 (adler32-zlib) [ 2372.281370] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 2372.538747] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 2374.264420] Key type lgssc registered [ 2375.512859] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2383.396340] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2383.423764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2385.071932] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2389.623519] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug +malloc all [ 2394.602123] LustreError: 75810:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2394.618801] LustreError: 75810:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2399.151706] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2399.583321] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2404.435693] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug +malloc all [ 2413.137264] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2419.618358] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2427.096262] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2427.554173] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2429.611268] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2435.938214] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug +malloc all [ 2445.990919] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2460.641287] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2460.654331] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2460.675487] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2460.689391] Lustre: Skipped 1 previous similar message [ 2462.574666] Lustre: server umount lustre-OST0000 complete [ 2470.882122] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2470.909903] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2470.919062] Lustre: Skipped 2 previous similar messages [ 2473.244334] Lustre: server umount lustre-MDT0000 complete [ 2476.008032] LustreError: 76538:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2476.030210] LustreError: 76538:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2477.210582] LustreError: 75791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789520437 with bad export cookie 8041042104995713474 [ 2477.219626] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2477.225022] LustreError: 75791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2477.532360] Lustre: server umount lustre-MDT0001 complete [ 2488.646777] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 2491.898882] Key type lgssc unregistered [ 2492.258501] LNet: 79343:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2492.264596] LNetError: 79343:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2493.296318] LNet: Removed LNI 192.168.204.107@tcp [ 2494.358948] Key type .llcrypt unregistered [ 2494.365980] Key type ._llcrypt unregistered [ 2519.052714] Key type ._llcrypt registered [ 2519.061263] Key type .llcrypt registered [ 2519.168810] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 2520.019753] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2520.076271] alg: No test for adler32 (adler32-zlib) [ 2521.107790] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 2521.304875] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 2522.951258] Key type lgssc registered [ 2524.163337] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2531.458781] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2531.481259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2547.105787] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2552.223356] LustreError: 80744:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.107@tcp: failed processing log, type 4: rc = -110 [ 2578.975583] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2585.454910] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2594.424935] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2604.741681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2604.809555] Lustre: *** cfs_fail_loc=706, val=0*** [ 2605.040650] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2605.089626] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2610.153482] LustreError: 80770:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2610.185762] LustreError: 80770:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2610.704202] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2613.228568] LustreError: 81829:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2613.267982] LustreError: 81829:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2618.341301] LustreError: 81830:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2618.363524] LustreError: 81830:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2620.269937] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2620.606272] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2625.424778] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2634.065253] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2640.603414] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2645.986445] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2645.995839] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2646.017608] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2646.516565] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2646.528052] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2646.540534] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2646.558181] Lustre: Skipped 1 previous similar message [ 2651.633671] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2656.742913] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2656.754340] Lustre: Skipped 1 previous similar message [ 2657.247174] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2657.370297] Lustre: server umount lustre-OST0000 complete [ 2661.856855] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2661.863565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2661.867749] Lustre: Skipped 1 previous similar message [ 2661.887277] Lustre: Skipped 2 previous similar messages [ 2667.508943] Lustre: server umount lustre-MDT0000 complete [ 2671.337296] LustreError: 80750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789520631 with bad export cookie 5536115132554147622 [ 2671.346570] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2671.359862] LustreError: 80750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2671.637611] Lustre: server umount lustre-MDT0001 complete [ 2682.228189] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 2684.645253] Key type lgssc unregistered [ 2684.921516] LNet: 84361:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2684.930762] LNetError: 84361:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2684.945541] LNet: Removed LNI 192.168.204.107@tcp [ 2685.769875] Key type .llcrypt unregistered [ 2685.773372] Key type ._llcrypt unregistered [ 2714.518550] Key type ._llcrypt registered [ 2714.520980] Key type .llcrypt registered [ 2714.649603] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 2715.872136] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2715.888700] alg: No test for adler32 (adler32-zlib) [ 2716.965069] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 2717.227348] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 2718.943214] Key type lgssc registered [ 2720.174157] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2728.733392] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2728.747884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2734.643267] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2743.579845] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2744.133707] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2749.423960] LustreError: 86464:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2749.438850] LustreError: 86464:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2749.561713] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2753.506575] LustreError: 86469:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2753.519821] LustreError: 86469:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2758.624576] LustreError: 86468:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2758.633595] LustreError: 86468:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2758.789366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2759.101065] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2764.280144] LustreError: 86462:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2764.305355] LustreError: 86462:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2766.789515] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2768.421489] LustreError: 86469:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2768.466192] LustreError: 86469:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2776.230982] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2776.258264] Lustre: MGS: Client d92c0600-5cb3-4e2a-8ab9-c7aa0f7c0451 (at 0@lo) reconnecting [ 2776.304717] LustreError: 86479:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2776.639385] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2776.722194] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2781.329701] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2798.560704] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2798.569852] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2798.587131] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2802.148892] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2802.152519] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2802.185896] Lustre: Skipped 1 previous similar message [ 2803.638854] Lustre: server umount lustre-OST0000 complete [ 2812.386081] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2812.386457] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2812.428645] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2812.438526] Lustre: Skipped 1 previous similar message [ 2813.739394] Lustre: server umount lustre-MDT0000 complete [ 2817.516866] LustreError: 86469:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2817.541482] LustreError: 86469:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2818.025665] Lustre: server umount lustre-MDT0001 complete [ 2822.686634] Lustre: server umount MGS complete [ 2838.162908] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 21:06:34 (1789520794) [ 2871.647472] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [ 2883.109147] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 2922.201444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2927.312770] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2937.508579] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2937.664167] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2937.721462] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2937.847360] Lustre: lustre-MDT0001: new disk, initializing [ 2937.982358] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2938.031039] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2938.040971] Lustre: 93656:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2938.055382] LustreError: 93656:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2943.611889] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2956.139556] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2956.482835] Lustre: lustre-OST0000: new disk, initializing [ 2956.490558] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2956.500570] Lustre: 94422:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2956.594419] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2964.003340] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2965.090599] Lustre: 94777:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000fb601515 x1876448433448704/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1789520940 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2973.785298] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2980.991875] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2981.002434] Lustre: MGS: Client c9478017-d123-4080-9dac-b7e1c163b1e9 (at 0@lo) reconnecting [ 2981.039542] Lustre: 92925:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2981.052282] LustreError: 93642:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2981.064558] LustreError: 93642:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2981.082606] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2981.157274] Lustre: lustre-MDT0000: new disk, initializing [ 2981.281639] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2981.325257] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2981.351119] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2982.369204] Lustre: 95491:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000ac5fc695 x1876448433455744/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1789520958 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2986.084892] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2987.491181] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2991.467609] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2991.474944] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2991.564858] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 3016.171787] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3016.173450] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3016.188823] Lustre: Skipped 2 previous similar messages [ 3021.214576] Lustre: server umount lustre-OST0000 complete [ 3026.401312] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3026.401846] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3026.406925] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3026.416735] Lustre: Skipped 1 previous similar message [ 3026.434558] Lustre: Skipped 2 previous similar messages [ 3030.587966] Lustre: server umount lustre-MDT0000 complete [ 3031.522265] LustreError: 93632:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3031.551282] LustreError: 93632:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3040.338884] Lustre: server umount lustre-MDT0001 complete [ 3043.857590] Lustre: server umount MGS complete [ 3055.603575] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 21:10:13 (1789521013) [ 3077.531861] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 3080.251202] Key type lgssc unregistered [ 3080.476432] LNet: 97990:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3080.482807] LNetError: 97990:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3080.504602] LNet: Removed LNI 192.168.204.107@tcp [ 3081.393487] Key type .llcrypt unregistered [ 3081.395936] Key type ._llcrypt unregistered [ 3101.927575] Key type ._llcrypt registered [ 3101.929807] Key type .llcrypt registered [ 3102.044932] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 3103.132462] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3103.211465] alg: No test for adler32 (adler32-zlib) [ 3104.279234] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 3104.531536] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 3106.256704] Key type lgssc registered [ 3107.363228] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3115.262819] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3115.279381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3116.650153] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3121.065124] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3126.243030] LustreError: 99208:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3126.257148] LustreError: 99208:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3130.019517] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3130.428290] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3135.130279] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3144.104326] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3150.673634] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3157.899815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3158.223176] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3161.330981] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3164.912423] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3173.317185] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3186.145367] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3186.156810] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3186.165061] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3186.177191] Lustre: Skipped 1 previous similar message [ 3189.855371] Lustre: server umount lustre-OST0000 complete [ 3196.385182] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3196.394043] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3196.397939] Lustre: Skipped 1 previous similar message [ 3196.413079] Lustre: Skipped 2 previous similar messages [ 3199.577738] Lustre: server umount lustre-MDT0000 complete [ 3201.505068] LustreError: 99208:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3201.520021] LustreError: 99208:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3202.706805] LustreError: 99187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789521162 with bad export cookie 5877831209509882813 [ 3202.710980] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3202.719669] LustreError: 99187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3203.002580] Lustre: server umount lustre-MDT0001 complete [ 3212.559890] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 3214.896615] Key type lgssc unregistered [ 3215.158877] LNet: 102741:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3215.166746] LNetError: 102741:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3215.185932] LNet: Removed LNI 192.168.204.107@tcp [ 3215.875799] Key type .llcrypt unregistered [ 3215.879193] Key type ._llcrypt unregistered [ 3231.703130] Key type ._llcrypt registered [ 3231.705036] Key type .llcrypt registered [ 3231.836939] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 3232.954677] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3233.008282] alg: No test for adler32 (adler32-zlib) [ 3234.102639] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 3234.363567] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 3236.066409] Key type lgssc registered [ 3237.280161] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3242.967581] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3242.973650] LustreError: 103824:0:(tgt_mount.c:2569:server_fill_super()) cfs_race id 716 sleeping [ 3243.011477] LustreError: 103831:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 waking [ 3243.016722] LustreError: 103824:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 awake: rc=4966 [ 3243.024290] LustreError: 103824:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3243.030449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3243.038832] LustreError: 103824:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3244.482781] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3251.855671] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3252.130208] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3256.349744] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3262.026374] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing lsmod [ 3264.748570] LustreError: 105034:0:(tgt_mount.c:2569:server_fill_super()) cfs_race id 716 sleeping [ 3264.754417] LustreError: 105036:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 waking [ 3264.765099] LustreError: 105034:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 awake: rc=4993 [ 3264.775134] LustreError: 105034:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3264.782909] LustreError: 105034:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3264.787472] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3265.278997] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3270.626044] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3270.631038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3270.635742] Lustre: Skipped 1 previous similar message [ 3275.744974] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3275.754352] Lustre: Skipped 3 previous similar messages [ 3276.771670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3276.782075] Lustre: Skipped 3 previous similar messages [ 3281.897265] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3281.901267] Lustre: Skipped 2 previous similar messages [ 3282.399917] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3282.587510] Lustre: server umount lustre-MDT0000 complete [ 3286.467153] LustreError: 103842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789521246 with bad export cookie 9535043787108912213 [ 3286.475478] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3286.477387] LustreError: 103842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3287.013289] LustreError: 103862:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3287.020515] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3287.032939] LustreError: 104298:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000006aad99c0 x1876448975541504/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3287.036961] LustreError: 103862:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3287.091394] Lustre: Skipped 1 previous similar message [ 3287.279537] Lustre: server umount lustre-MDT0001 complete [ 3293.601242] Lustre: server umount lustre-OST0000 complete [ 3302.043044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3302.523585] LustreError: 106101:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3302.541475] LustreError: 106101:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3302.616052] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3307.376423] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3308.001211] LustreError: 106102:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3312.111971] LustreError: 106101:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3316.715134] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3317.290851] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3321.986820] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3330.502465] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3337.028363] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3343.650361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3343.997577] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3348.094428] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3350.604638] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3357.791771] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3372.004779] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3372.016406] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3372.030982] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3373.026854] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3377.479672] Lustre: server umount lustre-OST0000 complete [ 3383.264866] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3383.281600] Lustre: Skipped 1 previous similar message [ 3383.286498] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3383.295396] Lustre: Skipped 3 previous similar messages [ 3387.597448] Lustre: server umount lustre-MDT0000 complete [ 3388.385933] LustreError: 106102:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3388.412727] LustreError: 106102:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3392.518704] LustreError: 106083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789521352 with bad export cookie 9535043787108912913 [ 3392.524990] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3392.542166] LustreError: 106083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3393.422598] Lustre: server umount lustre-MDT0001 complete [ 3408.627112] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 3411.435395] Key type lgssc unregistered [ 3411.706866] LNet: 109658:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3411.718677] LNetError: 109658:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3412.774713] LNet: Removed LNI 192.168.204.107@tcp [ 3413.683817] Key type .llcrypt unregistered [ 3413.687326] Key type ._llcrypt unregistered [ 3439.854699] Key type ._llcrypt registered [ 3439.856526] Key type .llcrypt registered [ 3439.945698] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 3441.206385] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3441.259087] alg: No test for adler32 (adler32-zlib) [ 3442.327466] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 3442.529375] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 3444.159247] Key type lgssc registered [ 3445.240388] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3454.294252] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3454.315406] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3455.858812] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3460.929636] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3465.186734] LustreError: 111072:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3465.200497] LustreError: 111072:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3470.310530] LustreError: 111073:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3471.847633] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3472.505759] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3477.460180] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3485.683836] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3492.185294] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3499.621389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3499.995430] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3501.046470] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3505.965591] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3512.533623] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3521.798732] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3524.489083] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3528.674387] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3528.674786] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3528.685080] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3528.701481] Lustre: Skipped 1 previous similar message [ 3533.491468] Lustre: server umount lustre-OST0000 complete [ 3538.918931] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3538.920385] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3538.926736] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3538.939344] Lustre: Skipped 2 previous similar messages [ 3543.271350] Lustre: server umount lustre-MDT0000 complete [ 3544.032530] LustreError: 112076:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3544.073112] LustreError: 112076:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3547.097912] LustreError: 111053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789521507 with bad export cookie 15899109005375002149 [ 3547.100074] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3547.107181] LustreError: 111053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3547.447847] Lustre: server umount lustre-MDT0001 complete [ 3558.806434] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 3561.549329] Key type lgssc unregistered [ 3561.827028] LNet: 114708:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3561.835319] LNetError: 114708:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3561.856590] LNet: Removed LNI 192.168.204.107@tcp [ 3562.803657] Key type .llcrypt unregistered [ 3562.806993] Key type ._llcrypt unregistered [ 3583.643517] Key type ._llcrypt registered [ 3583.649407] Key type .llcrypt registered [ 3583.810549] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 3585.012708] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3585.022443] alg: No test for adler32 (adler32-zlib) [ 3586.245221] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 3586.544068] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 3588.215361] Key type lgssc registered [ 3589.650757] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3598.942160] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3598.962891] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3600.637171] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3605.790380] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3611.104875] LustreError: 115929:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3611.114046] LustreError: 115929:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3614.616594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3614.930159] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3619.832330] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3628.311651] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3634.517794] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3642.517100] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3642.823437] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3646.987469] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3649.523253] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3657.432567] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3671.017947] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3671.024944] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3671.047622] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3674.597679] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3674.600679] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3674.620560] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3674.624062] Lustre: Skipped 1 previous similar message [ 3676.523872] Lustre: server umount lustre-OST0000 complete [ 3684.833379] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3684.844864] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3684.859918] Lustre: Skipped 1 previous similar message [ 3688.874254] Lustre: server umount lustre-MDT0000 complete [ 3689.972227] LustreError: 118225:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3690.008493] LustreError: 118225:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3695.073432] LustreError: 115925:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3695.114977] LustreError: 115925:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3695.533679] LustreError: 115908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789521655 with bad export cookie 14145270390965294426 [ 3695.542826] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3695.554900] LustreError: 115908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3697.027217] Lustre: server umount lustre-MDT0001 complete [ 3712.206997] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 3715.765801] Key type lgssc unregistered [ 3716.119762] LNet: 119483:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3716.135456] LNetError: 119483:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3716.161354] LNet: Removed LNI 192.168.204.107@tcp [ 3717.311594] Key type .llcrypt unregistered [ 3717.314220] Key type ._llcrypt unregistered [ 3742.131680] Key type ._llcrypt registered [ 3742.136052] Key type .llcrypt registered [ 3742.287181] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 3743.159314] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3743.175328] alg: No test for adler32 (adler32-zlib) [ 3744.331920] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 3744.637487] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 3746.367525] Key type lgssc registered [ 3747.680617] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3757.121649] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3757.140626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3759.048293] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3764.093203] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3768.296415] LustreError: 120899:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3768.310585] LustreError: 120899:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3773.419710] LustreError: 120900:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3773.933052] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3774.505845] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3780.200485] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3789.069693] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3796.118927] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3803.657383] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3804.140881] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3809.283044] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3811.259805] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3819.869988] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3826.300759] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3829.915516] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3829.922034] Lustre: Skipped 2 previous similar messages [ 3833.993254] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3834.049440] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3834.663604] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3834.669110] Lustre: Skipped 2 previous similar messages [ 3841.642425] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3841.650019] Lustre: Skipped 1 previous similar message [ 3849.358686] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.204.7@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3849.370254] Lustre: Skipped 2 previous similar messages [ 3855.466726] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.7@tcp 192.168.0.[2,10]@tcp [ 3859.662080] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.204.7/24@tcp in log lustre-MDT0000 [ 3859.668794] Lustre: Skipped 2 previous similar messages [ 3865.708730] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.7/24@tcp [ 3865.717897] Lustre: Skipped 1 previous similar message [ 3871.730182] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3871.745497] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3871.776704] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3875.817848] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3875.827889] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3875.861858] LustreError: Skipped 5 previous similar messages [ 3875.871121] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3877.672106] Lustre: server umount lustre-OST0000 complete [ 3886.052065] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3886.077118] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3886.079262] Lustre: Skipped 1 previous similar message [ 3888.613955] Lustre: server umount lustre-MDT0000 complete [ 3891.168634] LustreError: 120899:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3891.198070] LustreError: 120899:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3892.534709] LustreError: 120881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789521852 with bad export cookie 6138746569750812817 [ 3892.537724] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3892.545839] LustreError: 120881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3893.026223] Lustre: server umount lustre-MDT0001 complete [ 3905.616202] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 3908.364553] Key type lgssc unregistered [ 3908.704755] LNet: 125941:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3908.720333] LNetError: 125941:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3908.736538] LNet: Removed LNI 192.168.204.107@tcp [ 3909.610398] Key type .llcrypt unregistered [ 3909.613711] Key type ._llcrypt unregistered [ 3936.088628] Key type ._llcrypt registered [ 3936.090967] Key type .llcrypt registered [ 3936.186393] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 3937.108265] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3937.161654] alg: No test for adler32 (adler32-zlib) [ 3938.261372] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 3938.515066] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 3940.175256] Key type lgssc registered [ 3941.392964] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3949.945370] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3949.967805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3951.373580] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3951.394475] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3951.428559] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.204.7/24@tcp [ 3955.294415] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3961.831649] LustreError: 127500:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3961.855361] LustreError: 127500:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3963.592753] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3963.803925] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3963.835610] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.7/24@tcp [ 3963.881753] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3967.862763] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3975.620521] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3981.370929] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3987.856579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3988.445102] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3995.295216] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3996.678077] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 4005.073387] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4017.125341] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4017.125630] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4017.141140] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4017.150494] Lustre: Skipped 1 previous similar message [ 4017.167506] Lustre: Skipped 1 previous similar message [ 4022.242911] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4022.251949] Lustre: Skipped 1 previous similar message [ 4022.448933] Lustre: server umount lustre-OST0000 complete [ 4027.364551] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4027.366965] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4027.387780] Lustre: Skipped 1 previous similar message [ 4027.398234] Lustre: Skipped 1 previous similar message [ 4032.193829] Lustre: server umount lustre-MDT0000 complete [ 4032.480615] LustreError: 129345:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4032.501481] LustreError: 129345:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4035.255858] LustreError: 130047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789521995 with bad export cookie 11357964730631065325 [ 4035.258580] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4035.269687] LustreError: 130047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4035.549795] Lustre: server umount lustre-MDT0001 complete [ 4045.855994] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 4048.911118] Key type lgssc unregistered [ 4049.169572] LNet: 131086:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4049.175950] LNetError: 131086:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4049.196800] LNet: Removed LNI 192.168.204.107@tcp [ 4050.285541] Key type .llcrypt unregistered [ 4050.290536] Key type ._llcrypt unregistered [ 4077.452318] Key type ._llcrypt registered [ 4077.456378] Key type .llcrypt registered [ 4077.609487] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [ 4091.580304] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 4092.591096] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4092.621088] alg: No test for adler32 (adler32-zlib) [ 4093.687030] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 4093.908844] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 4095.552228] Key type lgssc registered [ 4096.733646] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4133.054526] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4133.095904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4134.504493] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4134.543291] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4134.638300] Lustre: lustre-MDT0000: new disk, initializing [ 4134.717804] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4134.737930] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4140.371381] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4152.748186] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4152.889284] Lustre: 134484:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4152.955390] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4152.960358] Lustre: Skipped 1 previous similar message [ 4153.047475] Lustre: lustre-MDT0001: new disk, initializing [ 4153.149914] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4153.197420] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4153.221983] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4158.867266] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4169.372349] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4176.240487] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4182.936531] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4183.196995] Lustre: lustre-OST0000: new disk, initializing [ 4183.200974] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4183.206718] Lustre: 136452:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4183.300789] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4189.517279] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4193.386330] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4193.395399] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4193.515353] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4199.521949] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4204.186862] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4204.376584] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4208.495289] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4208.679566] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4214.240747] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4214.242246] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4214.252419] LustreError: Skipped 1 previous similar message [ 4214.256893] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4214.295187] Lustre: Skipped 1 previous similar message [ 4216.778815] Lustre: server umount lustre-OST0000 complete [ 4224.483951] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4224.492114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4224.495588] Lustre: Skipped 1 previous similar message [ 4224.507265] Lustre: Skipped 2 previous similar messages [ 4227.647366] Lustre: server umount lustre-MDT0000 complete [ 4231.687440] LustreError: 136463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789522191 with bad export cookie 10766351396624535559 [ 4231.689399] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4231.701280] LustreError: 136463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4231.962522] Lustre: server umount lustre-MDT0001 complete [ 4240.321962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4240.869141] LustreError: 138669:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4240.888543] LustreError: 138669:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4240.964118] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4245.827826] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4245.984945] LustreError: 138670:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4251.104237] LustreError: 138669:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4254.685254] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4255.114116] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4259.401631] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4266.624504] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4272.811282] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4280.845582] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4281.502521] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4288.937378] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4299.144519] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4305.343708] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4305.623266] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4310.721201] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4310.964575] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4320.575522] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4320.782241] Lustre: lustre-OST0001: new disk, initializing [ 4320.785894] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4320.796636] Lustre: 142041:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4320.921483] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4327.532158] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4327.541236] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4327.721214] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4328.165359] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4336.467607] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4336.652407] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4342.899636] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4343.186290] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4349.765198] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8a0bc9afd800.ost_server_uuid 50 [ 4352.437477] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8a0bc9afd800.ost_server_uuid in IDLE state after 0 sec [ 4363.745987] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4363.762035] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4363.767501] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4363.795273] Lustre: Skipped 1 previous similar message [ 4363.822091] Lustre: Skipped 1 previous similar message [ 4368.594731] Lustre: server umount lustre-OST0001 complete [ 4368.887684] LustreError: 140493:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4368.911180] LustreError: 140493:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4373.988229] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4374.002472] LustreError: 140491:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4374.006339] LustreError: Skipped 1 previous similar message [ 4374.006384] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4374.012421] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4374.029742] LustreError: 140491:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4374.035747] Lustre: Skipped 1 previous similar message [ 4379.105659] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4379.120146] Lustre: Skipped 1 previous similar message [ 4384.242154] LustreError: 140494:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4384.258922] LustreError: 140494:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4387.295198] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4387.441328] Lustre: server umount lustre-OST0000 complete [ 4393.446676] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4393.456299] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4393.481228] Lustre: Skipped 1 previous similar message [ 4393.502751] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4393.512647] Lustre: Skipped 2 previous similar messages [ 4397.958882] Lustre: server umount lustre-MDT0000 complete [ 4402.155944] LustreError: 139404:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789522362 with bad export cookie 10766351396624536441 [ 4402.158171] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4402.164658] LustreError: 139404:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4402.539192] Lustre: server umount lustre-MDT0001 complete [ 4425.072108] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 4429.098549] Key type lgssc unregistered [ 4429.412242] LNet: 145058:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4429.429262] LNetError: 145058:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4430.505174] LNet: Removed LNI 192.168.204.107@tcp [ 4431.509932] Key type .llcrypt unregistered [ 4431.513420] Key type ._llcrypt unregistered [ 4486.148619] Key type ._llcrypt registered [ 4486.154165] Key type .llcrypt registered [ 4486.293638] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [ 4500.113812] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 4501.250066] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4501.291620] alg: No test for adler32 (adler32-zlib) [ 4502.355823] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 4502.652829] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 4504.328239] Key type lgssc registered [ 4505.491212] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4537.082834] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4537.109928] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4538.346971] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4538.401830] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4538.468506] Lustre: lustre-MDT0000: new disk, initializing [ 4538.541246] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4538.564483] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4543.057985] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4554.619788] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4554.718748] Lustre: 149211:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4554.778323] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4554.783205] Lustre: Skipped 1 previous similar message [ 4554.872395] Lustre: lustre-MDT0001: new disk, initializing [ 4554.990869] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4555.034092] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4555.050077] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4559.479701] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4568.982702] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4574.967571] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4581.655777] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4582.015656] Lustre: lustre-OST0000: new disk, initializing [ 4582.020874] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4582.032229] Lustre: 151180:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4582.106750] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4587.626221] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4587.641813] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4587.713631] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4587.878436] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4598.389670] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4604.643965] Lustre: Failing over lustre-OST0000 [ 4604.777560] Lustre: server umount lustre-OST0000 complete [ 4621.794202] Lustre: 147436:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789522565/real 1789522565] req@00000000f0ce9d30 x1876450305147904/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1789522581 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4621.837597] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4624.927205] Lustre: 147436:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789522568/real 1789522568] req@00000000e994c7ba x1876450305148416/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789522584 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4624.969857] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4626.785844] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4627.271144] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4627.288282] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4630.305202] Lustre: 147438:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789522573/real 1789522573] req@00000000a57bf494 x1876450305150080/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789522589 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4630.382616] Lustre: 147438:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4631.408164] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4634.158385] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4634.406102] Lustre: 147435:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789522578/real 1789522578] req@00000000b4c4d144 x1876450305151360/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789522594 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4634.445932] Lustre: 147435:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4634.552884] Lustre: lustre-OST0000: Recovery over after 0:03, of 3 clients 3 recovered and 0 were evicted. [ 4634.557375] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4638.469450] Lustre: Failing over lustre-MDT0000 [ 4638.782800] Lustre: server umount lustre-MDT0000 complete [ 4641.762908] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4641.784279] Lustre: Skipped 2 previous similar messages [ 4646.884728] LustreError: 149221:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4646.927499] LustreError: 149221:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 4647.486794] LustreError: 150520:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4652.002062] LustreError: 149216:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4652.028578] LustreError: 149216:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4655.016699] LustreError: 150011:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4655.027275] LustreError: 150011:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4657.764326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4657.913373] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4658.184971] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4658.248295] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4658.809849] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4663.283964] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4663.296576] Lustre: Skipped 1 previous similar message [ 4663.313135] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 4663.319967] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4663.364667] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4673.508901] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4673.513117] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4673.518530] Lustre: Skipped 1 previous similar message [ 4678.641646] LustreError: 152893:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4678.665106] LustreError: 152893:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4678.827665] Lustre: server umount lustre-OST0000 complete [ 4688.866309] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4688.867475] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4688.868789] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4688.868795] Lustre: Skipped 1 previous similar message [ 4688.900030] Lustre: Skipped 1 previous similar message [ 4690.528489] Lustre: server umount lustre-MDT0000 complete [ 4693.989786] LustreError: 150011:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4694.029255] LustreError: 150011:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4695.444296] LustreError: 151179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789522655 with bad export cookie 16627399787756655138 [ 4695.454885] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4695.473335] LustreError: 151179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4696.019687] Lustre: server umount lustre-MDT0001 complete [ 4712.558949] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 4715.801791] Key type lgssc unregistered [ 4716.260652] LNet: 154853:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4716.276259] LNetError: 154853:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4716.306176] LNet: Removed LNI 192.168.204.107@tcp [ 4717.476108] Key type .llcrypt unregistered [ 4717.479949] Key type ._llcrypt unregistered [ 4756.470706] Key type ._llcrypt registered [ 4756.475319] Key type .llcrypt registered [ 4756.632937] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 4757.746248] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4757.813106] alg: No test for adler32 (adler32-zlib) [ 4758.889370] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 4759.079434] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 4760.736253] Key type lgssc registered [ 4762.042618] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4770.356464] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4770.399478] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4771.892490] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4776.748858] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4782.050952] LustreError: 156373:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4782.083278] LustreError: 156373:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4787.176227] LustreError: 156374:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4788.467471] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4792.289260] LustreError: 156373:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4793.644090] LustreError: 156402:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4793.656505] LustreError: 156402:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4793.663390] LustreError: 156402:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 22, retries 0, failed: rc = -5 [ 4793.939583] Lustre: server umount lustre-MDT0000 complete [ 4809.494532] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [ 4818.155526] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 4854.534326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4854.779980] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4854.826706] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4854.886734] Lustre: lustre-MDT0000: new disk, initializing [ 4855.003137] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4855.019240] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4859.391262] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4869.937055] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4870.040908] Lustre: 159706:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4870.098403] LustreError: 159718:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4870.125537] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4870.127979] Lustre: Skipped 1 previous similar message [ 4870.189444] Lustre: lustre-MDT0001: new disk, initializing [ 4870.313308] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4870.345142] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4870.363702] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4875.932297] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4887.395427] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4894.186198] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4901.500547] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4901.783430] Lustre: lustre-OST0000: new disk, initializing [ 4901.786795] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4901.792785] Lustre: 161675:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4901.861972] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4907.580787] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4907.595851] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4907.677189] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4908.618309] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4917.743751] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4921.843635] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4923.655468] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 21:41:21 (1789522881) [ 5197.791937] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5197.793062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5197.807918] Lustre: Skipped 1 previous similar message [ 5201.372477] Lustre: server umount lustre-MDT0000 complete [ 5202.916120] LustreError: 162457:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5202.938631] LustreError: 162457:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5204.716492] LustreError: 159701:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789523164 with bad export cookie 5351593068450778049 [ 5204.720889] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5204.727903] LustreError: 159701:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5208.035047] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5208.051835] Lustre: Skipped 1 previous similar message [ 5208.061457] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5208.072073] Lustre: Skipped 2 previous similar messages [ 5214.181415] LustreError: 159714:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5214.189684] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5214.203863] LustreError: 159714:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5214.211914] Lustre: Skipped 1 previous similar message [ 5219.295737] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5219.296931] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5219.474919] Lustre: server umount lustre-MDT0001 complete [ 5229.256074] Lustre: server umount lustre-OST0000 complete [ 5238.226904] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [ 5247.666468] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 5278.116264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5278.356097] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5278.404791] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5278.464725] Lustre: lustre-MDT0000: new disk, initializing [ 5278.552508] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5278.577536] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5282.476499] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5292.280436] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5292.394672] Lustre: 166339:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5292.443323] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5292.446722] Lustre: Skipped 1 previous similar message [ 5292.504623] Lustre: lustre-MDT0001: new disk, initializing [ 5292.579524] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5292.622229] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5292.639658] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5296.825205] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5305.584611] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5310.851630] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5316.243971] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5316.473380] Lustre: lustre-OST0000: new disk, initializing [ 5316.476815] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5316.482718] Lustre: 168306:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5316.554312] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5321.443517] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5325.355414] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5325.362765] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5325.452299] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5328.903973] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5332.425168] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5332.582940] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5336.110478] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5336.282912] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5340.645940] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5340.650467] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5340.668724] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5341.157307] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5344.048869] Lustre: server umount lustre-OST0000 complete [ 5348.834780] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5348.842174] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5348.856146] Lustre: Skipped 1 previous similar message [ 5348.858515] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5348.861598] Lustre: Skipped 2 previous similar messages [ 5353.604578] Lustre: server umount lustre-MDT0000 complete [ 5356.513405] LustreError: 168677:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5356.524771] LustreError: 168677:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5356.780377] LustreError: 166329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789523316 with bad export cookie 5351593068450936886 [ 5356.782045] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5356.787667] LustreError: 166329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5357.045224] Lustre: server umount lustre-MDT0001 complete [ 5364.242939] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 21:48:42 (1789523322) [ 5377.247744] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [ 5385.199238] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 5417.980607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5418.172535] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5418.209829] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5418.283762] Lustre: lustre-MDT0000: new disk, initializing [ 5418.354737] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5418.371913] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5421.995296] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5430.990703] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5431.118262] Lustre: 173060:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5431.423373] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5435.263872] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5443.999382] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5449.242878] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5454.627536] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5454.838209] Lustre: lustre-OST0000: new disk, initializing [ 5454.840867] Lustre: Skipped 1 previous similar message [ 5454.845531] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5454.850440] Lustre: Skipped 2 previous similar messages [ 5454.857805] Lustre: 175034:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5460.327595] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5460.549821] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5460.563926] Lustre: Skipped 1 previous similar message [ 5460.571658] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5460.658188] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5468.504945] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5475.854872] Lustre: server umount lustre-OST0000 complete [ 5481.449728] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5481.454035] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5481.468403] Lustre: Skipped 2 previous similar messages [ 5481.488196] Lustre: Skipped 3 previous similar messages [ 5488.675752] Lustre: server umount lustre-MDT0000 complete [ 5491.472417] LustreError: 173051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789523451 with bad export cookie 5351593068450937768 [ 5491.476755] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5491.484234] LustreError: 173051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5502.055036] Lustre: server umount lustre-MDT0001 complete [ 5508.719927] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 21:51:06 (1789523466) [ 5523.688354] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [ 5532.237286] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 5564.732211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5564.944559] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5564.972660] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5565.035818] Lustre: lustre-MDT0000: new disk, initializing [ 5565.130192] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5565.135178] Lustre: Skipped 2 previous similar messages [ 5565.150037] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5569.529123] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5578.991641] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5579.155462] Lustre: 179360:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5579.361271] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5582.974664] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5591.749241] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5596.837444] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5602.478349] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5602.753229] Lustre: 181338:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5608.039874] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5612.071336] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5612.180684] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5615.730707] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5622.246979] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5622.256755] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5622.274481] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5622.281171] Lustre: Skipped 2 previous similar messages [ 5626.359994] Lustre: server umount lustre-OST0000 complete [ 5630.437729] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5630.445564] LustreError: Skipped 1 previous similar message [ 5644.255325] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5644.474887] Lustre: server umount lustre-MDT0000 complete [ 5647.330708] LustreError: 180271:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5647.333834] LustreError: 181336:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789523607 with bad export cookie 5351593068450938853 [ 5647.340075] LustreError: 180271:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5647.340132] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5647.348492] LustreError: 181336:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5647.581788] Lustre: server umount lustre-MDT0001 complete [ 5655.468546] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 5657.506993] Key type lgssc unregistered [ 5657.816495] LNet: 183192:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5657.823666] LNetError: 183192:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5657.841933] LNet: Removed LNI 192.168.204.107@tcp [ 5658.536189] Key type .llcrypt unregistered [ 5658.543263] Key type ._llcrypt unregistered [ 5677.240395] Key type ._llcrypt registered [ 5677.243044] Key type .llcrypt registered [ 5677.327053] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 5678.148967] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5678.163152] alg: No test for adler32 (adler32-zlib) [ 5679.163273] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 5679.321665] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 5680.967472] Key type lgssc registered [ 5681.792174] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5687.939986] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5687.964507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5689.287504] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5692.163246] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5698.216268] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5698.528977] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5701.957575] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5707.090945] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5711.044419] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5716.451553] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5716.725044] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5720.960640] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5726.038311] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5734.122089] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5754.852619] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5754.857556] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5754.859414] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5754.864643] Lustre: Skipped 1 previous similar message [ 5757.846799] Lustre: server umount lustre-OST0000 complete [ 5765.089220] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5765.091688] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5765.092105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5765.092110] Lustre: Skipped 1 previous similar message [ 5765.101075] Lustre: Skipped 1 previous similar message [ 5765.119992] LustreError: Skipped 1 previous similar message [ 5766.582178] Lustre: server umount lustre-MDT0000 complete [ 5769.248674] LustreError: 184565:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789523729 with bad export cookie 1181790651253607756 [ 5769.249447] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5769.259429] LustreError: 184565:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5769.499979] Lustre: server umount lustre-MDT0001 complete [ 5776.966483] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 5779.192567] Key type lgssc unregistered [ 5779.477732] LNet: 188137:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5779.485818] LNetError: 188137:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5779.502404] LNet: Removed LNI 192.168.204.107@tcp [ 5780.093744] Key type .llcrypt unregistered [ 5780.095769] Key type ._llcrypt unregistered [ 5797.473064] Key type ._llcrypt registered [ 5797.476248] Key type .llcrypt registered [ 5797.552309] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 5798.111446] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5798.221289] alg: No test for adler32 (adler32-zlib) [ 5799.167635] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 5799.340174] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 5800.991223] Key type lgssc registered [ 5801.744690] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5806.997254] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5807.008789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5808.325649] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5811.324448] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5817.213545] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5817.470250] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5820.624409] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5825.832836] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5829.940457] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5834.416674] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5834.658493] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5838.578876] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5843.521593] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5854.125429] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5858.272591] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5858.277628] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5858.288367] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5859.296546] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5859.300883] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5859.302039] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5859.330326] Lustre: Skipped 1 previous similar message [ 5864.050837] Lustre: server umount lustre-OST0000 complete [ 5869.040528] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8a0bd0549000.ost_server_uuid 50 [ 5870.296501] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5883.873139] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5883.883445] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5883.897644] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5883.902658] Lustre: Skipped 1 previous similar message [ 5885.921939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5885.933463] Lustre: Skipped 3 previous similar messages [ 5888.149752] Lustre: server umount lustre-MDT0000 complete [ 5890.936264] LustreError: 189507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789523850 with bad export cookie 1481383395164736409 [ 5890.940269] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5890.940812] LustreError: 189507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5891.163941] Lustre: server umount lustre-MDT0001 complete [ 5897.028659] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 21:57:34 (1789523854) [ 5901.412756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5901.690324] LustreError: 193464:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5901.704443] LustreError: 193464:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5901.744758] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5904.622304] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5906.920459] LustreError: 193465:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5910.093044] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5910.323063] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5912.754482] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5917.493205] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5920.983718] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5925.135402] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5925.403548] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5926.448661] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5929.078896] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5933.471486] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5937.323775] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5937.434232] Lustre: lustre-OST0001: new disk, initializing [ 5937.440266] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5937.448270] Lustre: 196302:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5939.061044] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5939.066052] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5939.114184] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5940.680102] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5946.511865] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5949.416211] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5949.425271] Lustre: Skipped 1 previous similar message [ 5949.426415] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5949.433814] Lustre: Skipped 1 previous similar message [ 5954.451138] Lustre: server umount lustre-OST0000 complete [ 5954.534191] LustreError: 195283:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5954.545921] LustreError: 195283:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5957.181732] LustreError: 195284:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5958.183405] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5958.324519] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5961.214698] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5961.368913] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5962.302264] LustreError: 195285:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5962.316182] LustreError: 195285:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5975.008677] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5975.010244] LustreError: 195285:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5975.015837] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5975.018107] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5975.029160] LustreError: 195285:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 5975.060391] Lustre: Skipped 2 previous similar messages [ 5979.813491] Lustre: server umount lustre-OST0001 complete [ 5985.255170] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5985.264734] Lustre: Skipped 2 previous similar messages [ 5988.867611] Lustre: server umount lustre-MDT0000 complete [ 5991.395039] LustreError: 193446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789523951 with bad export cookie 1481383395164737487 [ 5991.395972] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5991.401765] LustreError: 193446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5991.685740] Lustre: server umount lustre-MDT0001 complete [ 6007.598943] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 21:59:25 (1789523965) [ 6011.752446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6011.974609] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6012.002115] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6012.152111] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6012.156197] Lustre: Skipped 1 previous similar message [ 6014.708589] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6020.155664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6020.236431] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6020.273658] Lustre: 199736:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6023.703837] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6029.333385] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6033.649839] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6039.139709] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6039.300458] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6044.327295] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6046.858162] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6050.193861] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6055.586258] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6055.650701] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6055.823179] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6055.829248] Lustre: Skipped 2 previous similar messages [ 6060.547871] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6065.754516] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6067.681387] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6067.688515] LustreError: Skipped 1 previous similar message [ 6067.692507] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6067.707046] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6067.711264] Lustre: Skipped 2 previous similar messages [ 6073.725291] Lustre: server umount lustre-OST0000 complete [ 6074.853371] LustreError: 201587:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6074.871563] LustreError: 201587:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 6089.697056] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6092.931292] Lustre: server umount lustre-OST0001 complete [ 6100.448508] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6100.449552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6100.462666] Lustre: Skipped 4 previous similar messages [ 6100.474088] Lustre: Skipped 8 previous similar messages [ 6101.759363] Lustre: server umount lustre-MDT0000 complete [ 6104.330778] LustreError: 199727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789524064 with bad export cookie 1481383395164738887 [ 6104.337586] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6104.340124] LustreError: 199727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6104.583374] Lustre: server umount lustre-MDT0001 complete [ 6118.980698] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 22:01:16 (1789524076) [ 6129.578778] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [ 6136.089356] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 6162.278079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6162.435742] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6162.461665] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6162.531959] Lustre: lustre-MDT0000: new disk, initializing [ 6162.613791] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6162.627188] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6165.589913] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6172.965816] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6173.036555] Lustre: 207832:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6173.080273] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6173.083597] Lustre: Skipped 1 previous similar message [ 6173.144554] Lustre: lustre-MDT0001: new disk, initializing [ 6173.236826] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6173.243757] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6176.200975] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6182.604276] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6186.140158] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6190.540943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6190.727093] Lustre: lustre-OST0000: new disk, initializing [ 6190.731302] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6190.736562] Lustre: 209802:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6194.932758] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6197.289577] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6197.302368] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6197.381757] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6201.881102] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6205.427554] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6205.587921] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6209.035750] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6209.243677] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6212.576302] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6212.582817] LustreError: Skipped 2 previous similar messages [ 6212.585919] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6212.590636] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6212.593417] Lustre: Skipped 1 previous similar message [ 6212.603931] Lustre: Skipped 1 previous similar message [ 6216.851047] Lustre: server umount lustre-OST0000 complete [ 6234.079181] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6234.283226] Lustre: server umount lustre-MDT0000 complete [ 6236.956313] LustreError: 207825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789524196 with bad export cookie 1481383395164740448 [ 6236.959339] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6236.963347] LustreError: 207825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6237.201538] Lustre: server umount lustre-MDT0001 complete [ 6242.667222] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6242.920209] LustreError: 212019:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6242.934627] LustreError: 212019:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [ 6245.872193] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6251.459133] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6254.257320] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6258.673598] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6262.368809] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6266.805836] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6271.038773] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6275.742229] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6278.582832] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6278.728790] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6281.866583] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6282.004368] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6283.746904] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6283.753564] LustreError: Skipped 1 previous similar message [ 6289.599714] Lustre: server umount lustre-OST0000 complete [ 6293.237588] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6296.103234] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6322.428968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6322.637132] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6322.644178] Lustre: Skipped 5 previous similar messages [ 6326.765514] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6326.777959] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6326.809416] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6331.777979] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6336.995834] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6337.012507] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6343.699052] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 10 sec [ 6347.233692] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6347.240562] Lustre: Skipped 4 previous similar messages [ 6347.243813] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6347.247104] Lustre: Skipped 10 previous similar messages [ 6352.605428] Lustre: server umount lustre-OST0000 complete [ 6361.478995] Lustre: server umount lustre-MDT0000 complete [ 6363.911352] LustreError: 212002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789524323 with bad export cookie 1481383395164741330 [ 6363.916981] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6363.924662] LustreError: 212002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6364.128800] Lustre: server umount lustre-MDT0001 complete [ 6369.426976] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 22:05:27 (1789524327) [ 6373.889602] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6374.230730] LustreError: 218000:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6374.246457] LustreError: 218000:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6377.278427] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6382.857023] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6385.853096] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6390.739664] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6394.180844] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6398.085795] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6402.190275] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6406.863953] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6409.858275] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6409.989300] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6412.511759] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6412.628396] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6416.676285] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6416.818834] Lustre: lustre-OST0001: new disk, initializing [ 6416.821954] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6416.826062] Lustre: 221369:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6420.706014] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6424.613514] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6424.618567] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6424.656898] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6426.437135] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6429.186118] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6429.310719] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6432.044526] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6432.172588] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6434.787489] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6434.792149] LustreError: Skipped 4 previous similar messages [ 6439.510988] Lustre: server umount lustre-OST0001 complete [ 6443.002294] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6443.128609] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6445.723845] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6445.887429] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6472.347041] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6476.598646] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6477.802286] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6477.814247] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6477.818167] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6477.825591] Lustre: Skipped 1 previous similar message [ 6481.109084] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6482.060280] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6489.391231] Lustre: server umount lustre-OST0001 complete [ 6503.397092] LustreError: 219922:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6503.411379] LustreError: 219922:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 26 previous similar messages [ 6509.753962] Lustre: server umount lustre-OST0000 complete [ 6518.160700] Lustre: server umount lustre-MDT0000 complete [ 6520.582434] LustreError: 217982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789524480 with bad export cookie 1481383395164742289 [ 6520.585870] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6520.595203] LustreError: 217982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6526.825653] Lustre: server umount lustre-MDT0001 complete [ 6539.726553] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 22:08:17 (1789524497) [ 6542.983681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6543.156329] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6543.177445] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6545.674588] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6550.575308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6550.645857] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6550.664923] Lustre: 226604:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6553.186492] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6556.852075] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6559.835453] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6563.802377] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6563.909875] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6567.311337] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6571.441639] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6582.379481] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6582.431898] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6582.559892] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6582.563417] Lustre: Skipped 8 previous similar messages [ 6585.909024] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6589.433929] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6590.135267] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6592.174400] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6592.282689] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6594.525158] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6594.648304] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6597.580326] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8a0bd0bd1800.ost_server_uuid 50 [ 6598.389322] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8a0bd0bd1800.ost_server_uuid in FULL state after 0 sec [ 6598.950186] Lustre: Permanently deactivating lustre-OST0001 [ 6598.960809] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6600.446625] Lustre: Permanently reactivating lustre-OST0001 [ 6602.213837] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6602.225543] LustreError: Skipped 4 previous similar messages [ 6607.844530] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6607.847391] Lustre: Skipped 20 previous similar messages [ 6608.355845] Lustre: server umount lustre-OST0001 complete [ 6612.962577] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6612.968307] Lustre: Skipped 14 previous similar messages [ 6616.487191] Lustre: server umount lustre-OST0000 complete [ 6624.570517] Lustre: server umount lustre-MDT0000 complete [ 6626.301911] LustreError: 226597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789524586 with bad export cookie 1481383395164743584 [ 6626.306761] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6626.308040] LustreError: 226597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6626.456547] Lustre: server umount lustre-MDT0001 complete [ 6638.269120] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 22:09:56 (1789524596) [ 6641.756412] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6641.918048] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6641.932655] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6641.937745] Lustre: Skipped 5 previous similar messages [ 6644.196476] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6648.832184] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6648.899913] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6648.915070] Lustre: 233292:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6651.386769] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6655.311129] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6658.163498] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6661.638988] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6661.769826] Lustre: Permanently deactivating lustre-OST0000 [ 6661.776867] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6663.487538] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6664.465044] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6667.623192] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6670.673096] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6672.539307] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6673.436043] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6676.742850] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6677.505884] Lustre: Permanently reactivating lustre-OST0000 [ 6679.511033] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6679.516302] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6679.526598] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6679.551380] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 6679.561946] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6694.411383] Lustre: server umount lustre-OST0001 complete [ 6702.435425] Lustre: server umount lustre-OST0000 complete [ 6710.401944] Lustre: server umount lustre-MDT0000 complete [ 6712.184791] LustreError: 234034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789524672 with bad export cookie 1481383395164745243 [ 6712.189870] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6712.191834] LustreError: 234034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6712.369214] Lustre: server umount lustre-MDT0001 complete [ 6717.220824] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 6718.516473] Key type lgssc unregistered [ 6718.683453] LNet: 238144:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6718.690874] LNetError: 238144:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6718.703784] LNet: Removed LNI 192.168.204.107@tcp [ 6719.175456] Key type .llcrypt unregistered [ 6719.178036] Key type ._llcrypt unregistered [ 6730.619179] Key type ._llcrypt registered [ 6730.621447] Key type .llcrypt registered [ 6730.694251] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 6731.119065] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6731.127717] alg: No test for adler32 (adler32-zlib) [ 6732.052720] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 6732.205931] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 6733.847236] Key type lgssc registered [ 6734.517762] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6740.486883] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6740.504511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6741.788944] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6744.029844] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6748.868136] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6748.962991] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6748.986397] Lustre: 239574:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6748.997607] Lustre: Permanently deactivating lustre-MDT0001 [ 6749.006537] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6749.209379] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6751.476855] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6755.316194] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6755.437650] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6757.936137] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6761.284926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6761.504190] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6765.068223] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6766.628472] LustreError: 241427:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6766.633703] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 6766.641566] LustreError: 241427:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6769.504483] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6771.683923] LustreError: 241427:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6773.053506] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6773.185278] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6774.190981] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 6774.195750] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 6774.210339] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 6776.846324] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6781.391984] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6784.356140] Lustre: Permanently reactivating lustre-MDT0001 [ 6784.365835] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 6784.370322] Lustre: Skipped 2 previous similar messages [ 6788.501115] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6788.507972] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 6788.512954] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 6788.524606] LustreError: 243150:0:(llog_osd.c:2268:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 6788.525269] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6788.531910] LustreError: 243150:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 6788.540660] LustreError: 243150:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.107@tcp: cfg command failed: rc = -108 [ 6788.547366] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 6788.547366] [ 6788.551344] LustreError: 239577:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 6790.236546] Lustre: lustre-MDT0001: Client a6e92e8d-40a3-4b58-8796-7aa6addef74c (at 192.168.204.7@tcp) reconnecting [ 6793.304148] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6794.058382] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6796.919214] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6797.033816] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6799.047880] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6799.161345] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6800.353518] Lustre: Permanently deactivating lustre-MDT0001 [ 6800.361957] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6800.366500] Lustre: Skipped 2 previous similar messages [ 6809.122276] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6815.224208] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 6815.227636] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6815.237742] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6815.378110] Lustre: server umount lustre-MDT0000 complete [ 6817.313983] LustreError: 239564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789524777 with bad export cookie 5296322255477538086 [ 6817.321492] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6817.324440] LustreError: 239564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6817.494069] Lustre: server umount lustre-MDT0001 complete [ 6829.841344] Lustre: server umount lustre-OST0000 complete [ 6835.041687] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 22:13:13 (1789524793) [ 6835.679155] Lustre: 238799:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789524779/real 1789524779] req@00000000205478bc x1876452643506688/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789524795 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6835.691639] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6838.753234] Lustre: 238799:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789524782/real 1789524782] req@000000006db4fe3c x1876452643506944/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789524798 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6839.811708] Lustre: server umount lustre-OST0001 complete [ 6842.501383] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [ 6846.792713] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 6863.288515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6863.425984] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6863.432124] Lustre: Skipped 2 previous similar messages [ 6863.449399] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6863.496770] Lustre: lustre-MDT0000: new disk, initializing [ 6863.536945] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6863.552974] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6865.529208] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6871.556273] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6871.623740] Lustre: 247884:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6871.654427] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6871.657222] Lustre: Skipped 1 previous similar message [ 6871.711554] Lustre: lustre-MDT0001: new disk, initializing [ 6871.774465] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6871.794875] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6871.802779] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6873.977165] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6878.792899] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6881.264731] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6884.560846] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6884.702630] Lustre: lustre-OST0000: new disk, initializing [ 6884.705125] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6884.708862] Lustre: 249853:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6887.616923] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6892.486884] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6892.565392] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6892.576354] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6892.626493] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6893.629116] LustreError: 247892:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6898.798894] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6898.906745] Lustre: lustre-OST0001: new disk, initializing [ 6898.909914] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6898.914849] Lustre: 250963:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6898.969881] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6898.976969] Lustre: Skipped 1 previous similar message [ 6900.616387] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6900.621614] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6900.666312] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6901.635727] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6903.632114] LustreError: 247892:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6903.648444] LustreError: 249274:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6906.539875] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6907.211330] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6913.752300] LustreError: 249274:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6916.069821] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6916.075782] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6916.087301] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6916.093145] Lustre: Skipped 1 previous similar message [ 6917.192282] Lustre: lustre-OST0001: Not available for connect from 192.168.204.7@tcp (stopping) [ 6920.785596] Lustre: server umount lustre-OST0001 complete [ 6921.186396] LustreError: 249838:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6921.198782] LustreError: 249838:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6926.308208] LustreError: 250568:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6926.315399] LustreError: 250568:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6931.432325] LustreError: 249838:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6931.441336] LustreError: 249838:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6933.473084] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6933.473551] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6933.476384] LustreError: Skipped 2 previous similar messages [ 6933.483239] Lustre: Skipped 1 previous similar message [ 6933.486043] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6933.489740] Lustre: Skipped 1 previous similar message [ 6939.500588] Lustre: server umount lustre-OST0000 complete [ 6941.665068] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6941.667661] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6941.672422] Lustre: Skipped 1 previous similar message [ 6941.679745] Lustre: Skipped 3 previous similar messages [ 6947.453466] Lustre: server umount lustre-MDT0000 complete [ 6949.169262] LustreError: 247875:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789524909 with bad export cookie 5296322255477540907 [ 6949.174454] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6949.176780] LustreError: 247875:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6955.402668] Lustre: server umount lustre-MDT0001 complete [ 6959.837219] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 6961.214642] Key type lgssc unregistered [ 6961.381606] LNet: 252994:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6961.385848] LNetError: 252994:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6961.400759] LNet: Removed LNI 192.168.204.107@tcp [ 6961.798527] Key type .llcrypt unregistered [ 6961.800506] Key type ._llcrypt unregistered [ 6979.681115] Key type ._llcrypt registered [ 6979.682378] Key type .llcrypt registered [ 6979.735314] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 6980.303885] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6980.316521] alg: No test for adler32 (adler32-zlib) [ 6981.227919] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 6981.361155] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 6982.967435] Key type lgssc registered [ 6983.572320] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6987.485898] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6987.497496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6988.643652] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6988.669832] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6988.777747] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6990.750747] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6995.089281] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6995.169296] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6995.185149] Lustre: 255171:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6995.364896] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6997.478598] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7001.108190] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7003.912933] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7007.362282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7007.473951] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7007.564457] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7010.255117] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7011.554218] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7013.606065] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7057.894642] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7057.896159] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7057.896915] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7057.902973] Lustre: Skipped 1 previous similar message [ 7060.474137] Lustre: server umount lustre-OST0000 complete [ 7062.779193] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7069.202506] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7069.377290] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7072.224866] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7075.515038] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7077.607738] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7077.615533] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7077.615892] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7077.639223] Lustre: Skipped 1 previous similar message [ 7113.699031] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7113.699068] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7113.700519] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7113.700530] Lustre: Skipped 1 previous similar message [ 7113.702150] LustreError: Skipped 1 previous similar message [ 7113.716048] Lustre: Skipped 1 previous similar message [ 7115.619260] Lustre: server umount lustre-OST0000 complete [ 7118.305203] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7118.308525] LustreError: Skipped 1 previous similar message [ 7118.310475] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7118.316088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7118.318012] Lustre: Skipped 1 previous similar message [ 7123.299620] Lustre: server umount lustre-MDT0000 complete [ 7123.936506] LustreError: 255181:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7123.943201] LustreError: 255181:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 7124.754509] LustreError: 255164:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789525084 with bad export cookie 16061210505444275001 [ 7124.759183] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7124.759795] LustreError: 255164:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7124.904022] Lustre: server umount lustre-MDT0001 complete [ 7129.044161] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 7130.304921] Key type lgssc unregistered [ 7130.472774] LNet: 260591:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7130.480157] LNetError: 260591:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7130.492372] LNet: Removed LNI 192.168.204.107@tcp [ 7130.913369] Key type .llcrypt unregistered [ 7130.914672] Key type ._llcrypt unregistered [ 7141.774408] Key type ._llcrypt registered [ 7141.775993] Key type .llcrypt registered [ 7141.834880] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 7142.294653] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7142.363094] alg: No test for adler32 (adler32-zlib) [ 7143.252072] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 7143.372934] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 7145.003616] Key type lgssc registered [ 7145.663502] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7149.429464] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7149.439980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7150.687792] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7152.489244] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7157.053738] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7157.248191] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7159.262537] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7162.631602] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7165.015562] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7167.996325] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7168.176164] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7170.645527] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7173.629155] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7175.334563] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7196.128246] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7196.130862] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7196.140229] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7198.177909] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7198.178202] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7198.183054] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7199.454664] Lustre: server umount lustre-OST0000 complete [ 7203.296835] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7203.297315] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7203.297991] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7203.297995] Lustre: Skipped 1 previous similar message [ 7203.311269] Lustre: Skipped 1 previous similar message [ 7207.099695] Lustre: server umount lustre-MDT0000 complete [ 7208.417049] LustreError: 262709:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7208.423719] LustreError: 262709:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 7208.742747] LustreError: 261962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789525168 with bad export cookie 9061004091754156230 [ 7208.748066] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7208.753636] LustreError: 261962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7208.953361] Lustre: server umount lustre-MDT0001 complete [ 7213.699359] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 7215.131871] Key type lgssc unregistered [ 7215.285819] LNet: 266302:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7215.289438] LNetError: 266302:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7215.296698] LNet: Removed LNI 192.168.204.107@tcp [ 7215.711803] Key type .llcrypt unregistered [ 7215.713852] Key type ._llcrypt unregistered [ 7223.299126] Key type ._llcrypt registered [ 7223.300677] Key type .llcrypt registered [ 7223.342962] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 7223.705067] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7223.714595] alg: No test for adler32 (adler32-zlib) [ 7224.600730] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 7224.707774] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 7226.295168] Key type lgssc registered [ 7226.754126] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7230.099234] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7230.105795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7231.286063] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7232.851722] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7235.901883] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7236.051733] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7237.451506] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7240.035901] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7242.008822] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7244.386856] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7244.522498] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7246.861070] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7250.149225] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7256.803231] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7266.271990] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7266.274979] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7266.280419] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7267.297463] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7267.298202] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7267.299218] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7270.681838] Lustre: server umount lustre-OST0000 complete [ 7272.416342] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7272.417044] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7272.422291] Lustre: Skipped 1 previous similar message [ 7272.424705] Lustre: Skipped 1 previous similar message [ 7277.537087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7277.539595] Lustre: Skipped 1 previous similar message [ 7278.292971] Lustre: server umount lustre-MDT0000 complete [ 7279.836705] LustreError: 267521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789525239 with bad export cookie 17614481175048841270 [ 7279.840632] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7279.842728] LustreError: 267521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7279.985488] Lustre: server umount lustre-MDT0001 complete [ 7284.132732] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 7285.347882] Key type lgssc unregistered [ 7285.494505] LNet: 271191:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7285.497917] LNetError: 271191:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7285.506350] LNet: Removed LNI 192.168.204.107@tcp [ 7285.868548] Key type .llcrypt unregistered [ 7285.870065] Key type ._llcrypt unregistered [ 7295.979819] Key type ._llcrypt registered [ 7295.981564] Key type .llcrypt registered [ 7296.028618] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 7296.515154] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7296.579432] alg: No test for adler32 (adler32-zlib) [ 7297.456825] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 7297.557904] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 7299.151250] Key type lgssc registered [ 7299.560899] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7302.599703] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7302.606696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7303.782625] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7305.264951] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7308.332301] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7308.470819] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7310.053988] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7313.129535] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7315.470290] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7318.085392] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7318.212778] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7320.435317] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7323.288476] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7328.490757] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7340.000237] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7340.005135] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7340.015599] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7344.096843] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7344.097833] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7344.099156] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7344.112655] Lustre: Skipped 1 previous similar message [ 7345.238823] Lustre: server umount lustre-OST0000 complete [ 7349.219949] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7349.222713] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7349.230033] Lustre: Skipped 1 previous similar message [ 7349.234867] Lustre: Skipped 1 previous similar message [ 7354.336547] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7354.339836] Lustre: Skipped 1 previous similar message [ 7359.456471] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7359.459065] Lustre: Skipped 1 previous similar message [ 7360.991134] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7361.063100] Lustre: server umount lustre-MDT0000 complete [ 7362.556408] LustreError: 272560:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789525322 with bad export cookie 830666287768090299 [ 7362.559527] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7362.563375] LustreError: 272560:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7362.710703] Lustre: server umount lustre-MDT0001 complete [ 7366.852848] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 7367.975421] Key type lgssc unregistered [ 7368.105383] LNet: 276879:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7368.108673] LNetError: 276879:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7368.118371] LNet: Removed LNI 192.168.204.107@tcp [ 7368.463207] Key type .llcrypt unregistered [ 7368.464482] Key type ._llcrypt unregistered [ 7375.796776] Key type ._llcrypt registered [ 7375.798258] Key type .llcrypt registered [ 7375.843261] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 7376.200053] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7376.227269] alg: No test for adler32 (adler32-zlib) [ 7377.119974] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 7377.226122] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 7378.815203] Key type lgssc registered [ 7379.193128] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7381.955615] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7381.963709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7383.138361] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7384.544384] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7387.991832] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7388.152213] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7389.806333] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7392.742228] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7394.713103] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7397.182955] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7397.338694] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7399.407343] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7402.235448] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7408.617531] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 7418.849730] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7418.849982] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7418.851791] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7418.853803] Lustre: Skipped 1 previous similar message [ 7418.855984] LustreError: Skipped 1 previous similar message [ 7422.673444] Lustre: server umount lustre-OST0000 complete [ 7429.088887] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7429.088932] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7429.091688] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7429.095721] Lustre: Skipped 1 previous similar message [ 7429.099631] Lustre: Skipped 2 previous similar messages [ 7434.208477] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7434.210783] Lustre: Skipped 1 previous similar message [ 7438.303164] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7438.305071] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7438.310777] Lustre: Skipped 1 previous similar message [ 7438.364111] Lustre: server umount lustre-MDT0000 complete [ 7439.701188] LustreError: 278054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789525399 with bad export cookie 15356930977371208154 [ 7439.704602] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7439.707163] LustreError: 278054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7439.836288] Lustre: server umount lustre-MDT0001 complete [ 7443.598904] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 7444.685832] Key type lgssc unregistered [ 7444.808393] LNet: 281807:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7444.811587] LNetError: 281807:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7444.820405] LNet: Removed LNI 192.168.204.107@tcp [ 7445.149394] Key type .llcrypt unregistered [ 7445.150925] Key type ._llcrypt unregistered [ 7449.069155] Key type ._llcrypt registered [ 7449.070733] Key type .llcrypt registered [ 7449.108903] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 7454.822458] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [ 7461.010127] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 7461.331790] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7461.380603] alg: No test for adler32 (adler32-zlib) [ 7462.236256] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 7462.328973] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 7463.911197] Key type lgssc registered [ 7464.278752] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7475.999685] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7476.007879] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7477.088976] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7477.102555] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7477.141178] Lustre: lustre-MDT0000: new disk, initializing [ 7477.164953] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7477.172963] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7478.581827] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7482.779646] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7482.823686] Lustre: 285325:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7482.848199] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7482.850392] Lustre: Skipped 1 previous similar message [ 7482.885121] Lustre: lustre-MDT0001: new disk, initializing [ 7482.911948] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7482.924302] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7482.928330] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7484.216337] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7487.921869] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7489.725087] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7491.998326] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7492.128125] Lustre: lustre-OST0000: new disk, initializing [ 7492.130556] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7492.134321] Lustre: 287294:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7492.163961] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7494.139839] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7497.992886] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7499.425105] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7500.279948] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7500.283505] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7500.315290] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7500.517907] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 7501.892074] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7501.976715] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7505.376755] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7505.377455] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7505.380176] LustreError: Skipped 1 previous similar message [ 7505.384888] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7505.388025] Lustre: Skipped 1 previous similar message [ 7508.691716] Lustre: server umount lustre-OST0000 complete [ 7510.496794] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7510.497331] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7510.500636] Lustre: Skipped 2 previous similar messages [ 7510.504375] Lustre: Skipped 1 previous similar message [ 7515.616439] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7515.618246] Lustre: Skipped 1 previous similar message [ 7520.737100] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7520.740162] Lustre: Skipped 1 previous similar message [ 7524.319150] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7524.379108] Lustre: server umount lustre-MDT0000 complete [ 7525.657436] LustreError: 285318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789525485 with bad export cookie 14339588762409198104 [ 7525.661240] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7525.662617] LustreError: 285318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7525.781364] Lustre: server umount lustre-MDT0001 complete [ 7528.723610] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 22:24:47 (1789525487) [ 7530.993485] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7531.174175] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7532.568045] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7535.412887] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7535.532800] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7536.905789] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7539.358942] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7541.214637] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7543.509756] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7543.618564] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7545.466886] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7547.838812] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7561.184156] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7561.186810] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7561.191280] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7562.210573] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7566.989616] Lustre: server umount lustre-OST0000 complete [ 7571.424076] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7571.427115] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7571.432107] Lustre: Skipped 1 previous similar message [ 7571.434365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7571.436407] Lustre: Skipped 2 previous similar messages [ 7574.354068] Lustre: server umount lustre-MDT0000 complete [ 7575.510037] LustreError: 289734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789525535 with bad export cookie 14339588762409198972 [ 7575.511509] LustreError: 290445:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000bd2:0x1:0x0]: rc = -5 [ 7575.513308] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7575.515113] LustreError: 289734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7575.623769] Lustre: server umount lustre-MDT0001 complete [ 7579.076636] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 7580.102388] Key type lgssc unregistered [ 7580.245438] LNet: 293256:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7580.247499] LNetError: 293256:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7580.256381] LNet: Removed LNI 192.168.204.107@tcp [ 7580.610632] Key type .llcrypt unregistered [ 7580.611852] Key type ._llcrypt unregistered [ 7589.468197] Key type ._llcrypt registered [ 7589.469189] Key type .llcrypt registered [ 7589.512701] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [ 7595.584335] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 7595.945363] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7595.989405] alg: No test for adler32 (adler32-zlib) [ 7596.838896] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 7596.925395] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 7598.503176] Key type lgssc registered [ 7598.860824] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7613.926306] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7613.934657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7615.025977] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7615.040612] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7615.082437] Lustre: lustre-MDT0000: new disk, initializing [ 7615.113788] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7615.122569] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7616.571949] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7620.900224] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7620.955046] Lustre: 297048:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7620.984085] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7620.986748] Lustre: Skipped 1 previous similar message [ 7621.027525] Lustre: lustre-MDT0001: new disk, initializing [ 7621.067575] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7621.084149] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7621.087929] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7622.532504] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7626.362213] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7628.281781] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7630.624965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7630.747329] Lustre: lustre-OST2710: new disk, initializing [ 7630.749546] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 7630.752326] Lustre: 299014:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7630.791744] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 7632.890603] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7636.649141] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7637.501249] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 7637.505511] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 7637.547021] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 7638.974222] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7639.036628] Lustre: lustre-OST03e8: new disk, initializing [ 7639.038750] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 7639.041775] Lustre: 300077:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7639.073812] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 7640.990581] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7643.493028] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7643.653879] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 7643.658810] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 7643.692869] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 7646.208572] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 7646.286213] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7647.687551] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 7647.767159] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7649.120499] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 7649.196146] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7650.549930] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 7650.635888] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7651.808236] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7651.811320] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7651.818535] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7653.859207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7653.859643] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7653.861985] Lustre: Skipped 1 previous similar message [ 7653.866400] Lustre: Skipped 1 previous similar message [ 7657.823863] Lustre: server umount lustre-MDT0000 complete [ 7659.127086] LustreError: 297039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789525619 with bad export cookie 3649108362815529116 [ 7659.130590] LustreError: 297039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7659.130695] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7659.248507] Lustre: server umount lustre-MDT0001 complete [ 7670.269249] Lustre: server umount lustre-OST2710 complete [ 7681.761239] Lustre: server umount lustre-OST03e8 complete [ 7687.929593] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [ 7691.281332] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 7702.899285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7702.979980] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7702.991928] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7703.031195] Lustre: lustre-MDT0000: new disk, initializing [ 7703.052506] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7703.060452] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7704.500719] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7708.717226] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7708.762283] Lustre: 305312:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7708.858975] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7710.237798] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7714.197113] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7716.049079] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7718.273691] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7718.371492] Lustre: lustre-OST0000: new disk, initializing [ 7718.372847] Lustre: Skipped 1 previous similar message [ 7718.374451] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7718.375967] Lustre: Skipped 2 previous similar messages [ 7718.378033] Lustre: 307282:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7718.400279] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7718.403053] Lustre: Skipped 1 previous similar message [ 7720.175909] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7723.797220] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7724.532135] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7724.534776] Lustre: Skipped 1 previous similar message [ 7724.536487] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7724.566048] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7725.121361] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7725.193094] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7726.545946] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7726.621390] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7729.634263] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7729.635194] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7729.637355] LustreError: Skipped 1 previous similar message [ 7729.642766] Lustre: Skipped 1 previous similar message [ 7729.645198] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7729.648399] Lustre: Skipped 3 previous similar messages [ 7733.388272] Lustre: server umount lustre-OST0000 complete [ 7734.753724] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7734.753852] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7734.754490] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7734.758608] Lustre: Skipped 1 previous similar message [ 7739.872251] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7739.874137] Lustre: Skipped 2 previous similar messages [ 7740.760491] Lustre: server umount lustre-MDT0000 complete [ 7742.073425] LustreError: 306110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789525701 with bad export cookie 3649108362815530796 [ 7742.077754] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7742.081527] LustreError: 306110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7742.196227] Lustre: server umount lustre-MDT0001 complete [ 7745.035524] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 22:28:23 (1789525703) [ 7745.546827] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 7746.123942] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 22:28:24 (1789525704) [ 7747.171514] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 7755.910483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7755.983939] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7755.990338] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7756.047219] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7757.333470] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7760.190338] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7760.222541] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7760.234976] Lustre: 311131:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7761.678541] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7777.569899] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7777.631976] Lustre: Denying initial registration attempt from nid 192.168.204.107@tcp, specified as failover [ 7777.634574] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 7777.637097] LustreError: 312583:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -99 [ 7777.639165] LustreError: 312583:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 7777.640830] LustreError: 312583:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 7777.662408] Lustre: server umount lustre-OST0000 complete [ 7777.663623] LustreError: 312583:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 7780.831780] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7780.835126] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7780.839493] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7794.143840] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7794.195845] Lustre: server umount lustre-MDT0000 complete [ 7795.448177] LustreError: 311122:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789525755 with bad export cookie 3649108362815531664 [ 7795.451186] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7795.452457] LustreError: 311122:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7795.568696] Lustre: server umount lustre-MDT0001 complete [ 7799.258578] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 7800.310959] Key type lgssc unregistered [ 7800.448434] LNet: 313703:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7800.451252] LNetError: 313703:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7800.459413] LNet: Removed LNI 192.168.204.107@tcp [ 7800.827467] Key type .llcrypt unregistered [ 7800.829269] Key type ._llcrypt unregistered [ 7804.774935] Key type ._llcrypt registered [ 7804.776537] Key type .llcrypt registered [ 7804.820288] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 7805.230959] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7805.282524] alg: No test for adler32 (adler32-zlib) [ 7806.136412] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 7806.228765] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 7807.823211] Key type lgssc registered [ 7808.228089] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7833.140669] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 7836.755488] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7836.762658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7837.853104] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7837.862457] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7837.917023] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7839.358859] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7842.549216] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7842.586664] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7842.598764] Lustre: 316438:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7842.713169] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7844.222940] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7847.061241] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7849.205669] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7864.802219] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7864.881600] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7864.960546] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7866.974061] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7869.547983] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7870.391125] Lustre: server umount lustre-OST0000 complete [ 7873.503902] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7873.507575] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7873.513310] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7875.552621] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7875.552683] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7875.559822] Lustre: Skipped 1 previous similar message [ 7877.844149] Lustre: server umount lustre-MDT0000 complete [ 7879.095353] LustreError: 316428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789525839 with bad export cookie 16674172966186277878 [ 7879.099801] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7879.100194] LustreError: 316428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7879.218631] Lustre: server umount lustre-MDT0001 complete [ 7882.940262] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 7883.962735] Key type lgssc unregistered [ 7884.095276] LNet: 320002:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7884.097503] LNetError: 320002:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7884.107489] LNet: Removed LNI 192.168.204.107@tcp [ 7884.429931] Key type .llcrypt unregistered [ 7884.431435] Key type ._llcrypt unregistered [ 7893.913622] Key type ._llcrypt registered [ 7893.915209] Key type .llcrypt registered [ 7893.958847] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 7894.307945] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7894.373448] alg: No test for adler32 (adler32-zlib) [ 7895.244983] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 7895.342359] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 7896.927237] Key type lgssc registered [ 7897.294580] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7900.184516] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7900.193899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7901.392250] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7902.867524] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7905.911379] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7906.035573] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7907.393878] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7909.853326] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7911.737530] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7915.937967] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7916.045892] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7917.925399] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7920.401966] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7935.457084] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7935.459066] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7935.461577] Lustre: Skipped 2 previous similar messages [ 7935.464028] Lustre: Skipped 1 previous similar message [ 7937.090199] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.7@tcp (stopping) [ 7940.050755] Lustre: server umount lustre-MDT0000 complete [ 7940.577157] LustreError: 321387:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7940.582404] LustreError: 321387:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 7941.347949] LustreError: 321371:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789525901 with bad export cookie 8475375733429771033 [ 7941.349861] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7941.351258] LustreError: 321371:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7942.202614] LustreError: 323878:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7942.208986] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.7@tcp (stopping) [ 7945.695792] LustreError: 321415:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7945.696060] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7945.705957] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7947.504250] Lustre: server umount lustre-MDT0001 complete [ 7949.813200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 7952.671632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7966.175577] LustreError: 320701:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000970fdcee x1876453862957312/t0(0) o250->MGC192.168.204.107@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7966.292315] LustreError: 323222:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7966.299703] LustreError: 323222:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7966.316751] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7966.339327] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [ 7967.658481] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7970.494827] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7970.619277] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7971.683530] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7972.013554] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7974.547761] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7976.803980] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7979.178693] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 7981.130758] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7981.674106] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7986.144081] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7986.146437] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7986.151485] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7986.153590] Lustre: Skipped 2 previous similar messages [ 7987.171405] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7989.073339] Lustre: server umount lustre-OST0000 complete [ 7991.263699] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7991.266179] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7991.271971] Lustre: Skipped 1 previous similar message [ 7996.443885] Lustre: server umount lustre-MDT0000 complete [ 7997.408308] LustreError: 325003:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7997.414198] LustreError: 325003:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7997.738555] LustreError: 321371:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789525957 with bad export cookie 8475375733429778159 [ 7997.741703] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7997.742786] LustreError: 321371:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7997.845890] Lustre: server umount lustre-MDT0001 complete [ 8001.819363] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 8002.947063] Key type lgssc unregistered [ 8003.083512] LNet: 327669:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8003.086400] LNetError: 327669:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8003.093355] LNet: Removed LNI 192.168.204.107@tcp [ 8003.429508] Key type .llcrypt unregistered [ 8003.430946] Key type ._llcrypt unregistered [ 8013.320608] Key type ._llcrypt registered [ 8013.322124] Key type .llcrypt registered [ 8013.363897] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 8013.813548] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8013.831277] alg: No test for adler32 (adler32-zlib) [ 8014.695975] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 8014.802039] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 8016.391179] Key type lgssc registered [ 8016.775168] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8019.705328] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8019.712372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8020.851272] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8022.263955] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8025.285200] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8025.424387] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8026.884450] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8029.419773] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8031.342316] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8035.424760] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8035.430590] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8035.433032] Lustre: Skipped 1 previous similar message [ 8038.479400] Lustre: server umount lustre-MDT0000 complete [ 8039.785383] LustreError: 330672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789525999 with bad export cookie 6788301561234283024 [ 8039.788741] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8039.790942] LustreError: 330672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8039.916630] Lustre: server umount lustre-MDT0001 complete [ 8045.028504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8045.132695] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8045.143673] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8045.214593] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8046.607951] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8049.487546] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8049.523369] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8049.534067] Lustre: 331644:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8049.635347] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8051.020375] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8053.385041] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8055.183678] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8059.745993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8059.809095] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 8059.812582] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 8059.815562] LustreError: 333525:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 8059.818493] LustreError: 333525:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 8059.821430] LustreError: 333525:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 8059.845177] Lustre: server umount lustre-OST0000 complete [ 8059.846632] LustreError: 333525:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 8064.674525] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8064.766744] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8064.843720] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8066.729826] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8069.165212] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8071.839553] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8071.873465] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 8071.911049] Lustre: lustre-OST0001: new disk, initializing [ 8071.913830] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8071.917330] Lustre: 335054:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8071.972842] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [ 8073.893807] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8076.298122] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8077.271090] Lustre: server umount lustre-OST0001 complete [ 8077.282356] LustreError: 333991:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8077.288045] LustreError: 333991:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 8082.401137] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8082.402391] LustreError: 333992:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8082.402830] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8082.402836] Lustre: Skipped 1 previous similar message [ 8082.403781] LustreError: Skipped 1 previous similar message [ 8082.404033] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8082.409042] LustreError: 333992:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8084.622922] Lustre: server umount lustre-OST0000 complete [ 8087.520664] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8087.521493] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8087.524623] Lustre: Skipped 2 previous similar messages [ 8087.526673] Lustre: Skipped 1 previous similar message [ 8092.640287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8092.642325] Lustre: Skipped 2 previous similar messages [ 8097.760271] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8097.762600] Lustre: Skipped 1 previous similar message [ 8100.319121] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8100.378074] Lustre: server umount lustre-MDT0000 complete [ 8101.576381] LustreError: 331634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789526061 with bad export cookie 6788301561234283563 [ 8101.579869] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8101.580660] LustreError: 331634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8101.689430] Lustre: server umount lustre-MDT0001 complete [ 8105.262726] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 8106.235546] Key type lgssc unregistered [ 8106.366345] LNet: 337010:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8106.368927] LNetError: 337010:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8106.377282] LNet: Removed LNI 192.168.204.107@tcp [ 8106.680688] Key type .llcrypt unregistered [ 8106.682194] Key type ._llcrypt unregistered [ 8136.748864] Key type ._llcrypt registered [ 8136.749835] Key type .llcrypt registered [ 8136.785404] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [ 8142.804645] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 8143.165833] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8143.205483] alg: No test for adler32 (adler32-zlib) [ 8144.056746] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 8144.143572] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 8145.719119] Key type lgssc registered [ 8146.053225] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8156.817390] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8156.823297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8157.909606] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8157.921733] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8157.955326] Lustre: lustre-MDT0000: new disk, initializing [ 8157.977984] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8157.990249] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8159.304536] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8163.393278] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8163.433189] Lustre: 341906:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8163.457156] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8163.459490] Lustre: Skipped 1 previous similar message [ 8163.493254] Lustre: lustre-MDT0001: new disk, initializing [ 8163.525571] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8163.539658] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8163.543426] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8164.870836] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8168.572894] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8170.350263] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8172.521603] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8172.610103] Lustre: lustre-OST0000: new disk, initializing [ 8172.611881] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8172.614291] Lustre: 343873:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8172.641289] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8174.326828] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8174.329632] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8174.362950] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8174.441700] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8177.992119] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8179.303974] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8179.368286] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8180.688533] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8180.758758] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8184.803741] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8184.805525] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8184.806261] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8184.806664] LustreError: Skipped 1 previous similar message [ 8184.815555] Lustre: Skipped 1 previous similar message [ 8187.472953] Lustre: server umount lustre-OST0000 complete [ 8189.408081] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8189.410699] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8189.416770] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8189.419112] Lustre: Skipped 1 previous similar message [ 8195.040618] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8195.044417] Lustre: Skipped 2 previous similar messages [ 8200.163598] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8200.171316] Lustre: Skipped 1 previous similar message [ 8203.232053] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8203.384102] Lustre: server umount lustre-MDT0000 complete [ 8206.060620] LustreError: 341899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789526165 with bad export cookie 8704935591417217917 [ 8206.066899] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8206.069447] LustreError: 341899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8206.324936] Lustre: server umount lustre-MDT0001 complete [ 8212.847028] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 22:36:10 (1789526170) [ 8218.534803] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 22:36:16 (1789526176) [ 8224.825590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8225.137108] LustreError: 346628:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8225.155774] LustreError: 346628:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 8225.196525] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8228.655498] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8230.406994] LustreError: 346629:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8235.492131] LustreError: 346628:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8239.521605] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8240.302406] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8247.303739] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8257.132792] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8262.081091] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8266.317091] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8266.554954] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8270.096869] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8274.821905] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8289.995579] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 8290.971840] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 8291.864097] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 8292.849359] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 8293.898556] Lustre: Failing over lustre-MDT0000 [ 8294.034854] Lustre: server umount lustre-MDT0000 complete [ 8296.419183] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8296.419289] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8296.422822] LustreError: 346623:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8296.422833] LustreError: 346623:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8296.428364] Lustre: Skipped 3 previous similar messages [ 8301.536868] LustreError: 347356:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8301.552868] LustreError: 347356:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 8309.260897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8309.352422] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8309.520797] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8309.562028] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 8312.250505] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8312.891315] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 8314.860362] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8314.878431] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 8314.902660] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 8318.054342] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8319.072439] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8321.657785] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 8322.873385] Lustre: Failing over lustre-MDT0000 [ 8323.042806] Lustre: server umount lustre-MDT0000 complete [ 8323.150952] LustreError: 348925:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8323.167062] LustreError: 348925:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 8325.089298] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8325.095334] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8325.105633] Lustre: Skipped 2 previous similar messages [ 8337.974477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8338.051673] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8338.230193] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 8338.492055] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 8340.638264] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8343.527282] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8343.533574] Lustre: Skipped 2 previous similar messages [ 8343.550607] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 8343.574221] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 8346.462085] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8347.351182] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8353.765330] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8353.766608] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8353.779982] Lustre: Skipped 1 previous similar message [ 8353.789455] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8356.275242] Lustre: server umount lustre-MDT0000 complete [ 8357.971298] LustreError: 346610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789526317 with bad export cookie 8704935591417220612 [ 8357.975260] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8357.976321] LustreError: 346610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8358.126717] Lustre: server umount lustre-MDT0001 complete [ 8370.442209] Lustre: server umount lustre-OST0000 complete [ 8378.895804] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8379.180731] LustreError: 352983:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8379.195556] LustreError: 352983:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [ 8379.221597] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8379.227025] Lustre: Skipped 1 previous similar message [ 8381.416565] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8386.081543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8388.749274] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8393.281976] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8396.550939] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8400.288750] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8403.740291] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8404.588238] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 8408.121278] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8415.200504] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8415.204812] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8415.211488] Lustre: Skipped 1 previous similar message [ 8415.214133] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8415.216809] Lustre: Skipped 2 previous similar messages [ 8421.190738] Lustre: server umount lustre-OST0000 complete [ 8429.334825] Lustre: server umount lustre-MDT0000 complete [ 8431.417259] LustreError: 354825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789526391 with bad export cookie 8704935591417221361 [ 8431.418066] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8431.425712] LustreError: 354825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8431.615443] Lustre: server umount lustre-MDT0001 complete [ 8437.909506] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 8439.611746] Key type lgssc unregistered [ 8439.781293] LNet: 356512:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8439.790913] LNetError: 356512:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8439.807754] LNet: Removed LNI 192.168.204.107@tcp [ 8440.281920] Key type .llcrypt unregistered [ 8440.283993] Key type ._llcrypt unregistered [ 8455.597584] Key type ._llcrypt registered [ 8455.600730] Key type .llcrypt registered [ 8455.675728] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 8456.402419] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8456.412902] alg: No test for adler32 (adler32-zlib) [ 8457.391481] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 8457.551809] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 8459.175302] Key type lgssc registered [ 8459.862308] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8464.299258] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8464.318616] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8465.636674] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8467.982446] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8472.587862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8472.826576] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8474.936836] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8478.881661] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8482.239168] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8485.896764] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8486.133438] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8489.390584] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8490.221814] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 8493.393896] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8503.777148] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8503.783787] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8503.795209] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8507.875865] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8507.876224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8507.889859] Lustre: Skipped 1 previous similar message [ 8507.893066] Lustre: Skipped 1 previous similar message [ 8509.082874] Lustre: server umount lustre-MDT0000 complete [ 8511.328290] LustreError: 357982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789526471 with bad export cookie 6805849794169677937 [ 8511.333165] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8511.334846] LustreError: 357982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8511.568936] Lustre: server umount lustre-MDT0001 complete [ 8523.259371] Lustre: server umount lustre-OST0000 complete [ 8530.626376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8530.948453] LustreError: 361710:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8530.961283] LustreError: 361710:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8530.999140] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8533.195894] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8536.034687] LustreError: 361710:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8537.870395] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8538.109985] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8540.240433] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8543.699120] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8546.516732] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8550.208383] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8550.406900] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8553.612681] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8555.562244] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 8557.334551] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8567.783574] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8567.784846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8567.791239] Lustre: Skipped 2 previous similar messages [ 8567.794965] Lustre: Skipped 1 previous similar message [ 8571.537344] Lustre: server umount lustre-MDT0000 complete [ 8572.898087] LustreError: 361711:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8572.905119] LustreError: 361711:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8573.524843] LustreError: 363552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789526533 with bad export cookie 6805849794169679050 [ 8573.529470] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8573.540601] LustreError: 363552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8573.699381] Lustre: server umount lustre-MDT0001 complete [ 8585.289804] Lustre: server umount lustre-OST0000 complete [ 8592.780641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8593.030348] LustreError: 365618:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8593.067931] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8594.868349] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8598.497605] LustreError: 365619:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8598.569103] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8600.547269] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8603.808830] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8606.330728] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8609.654313] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8612.490804] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8615.949510] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8618.021600] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 8628.194091] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8628.194775] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8628.196177] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8628.196181] Lustre: Skipped 1 previous similar message [ 8628.200982] Lustre: Skipped 1 previous similar message [ 8629.409331] Lustre: server umount lustre-OST0000 complete [ 8633.313810] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8633.314766] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8633.321195] Lustre: Skipped 1 previous similar message [ 8633.325263] Lustre: Skipped 1 previous similar message [ 8637.377917] Lustre: server umount lustre-MDT0000 complete [ 8638.433711] LustreError: 365619:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8638.451337] LustreError: 365619:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8639.297023] LustreError: 365598:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789526599 with bad export cookie 6805849794169680058 [ 8639.301977] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8639.304267] LustreError: 365598:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8639.464036] Lustre: server umount lustre-MDT0001 complete [ 8644.508574] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 8645.819254] Key type lgssc unregistered [ 8645.984430] LNet: 369147:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8645.990364] LNetError: 369147:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8646.000882] LNet: Removed LNI 192.168.204.107@tcp [ 8646.415517] Key type .llcrypt unregistered [ 8646.417515] Key type ._llcrypt unregistered [ 8658.096806] Key type ._llcrypt registered [ 8658.102376] Key type .llcrypt registered [ 8658.163137] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 8658.568462] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8658.637063] alg: No test for adler32 (adler32-zlib) [ 8659.540448] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 8659.676747] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 8661.303308] Key type lgssc registered [ 8661.790592] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8665.059475] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8665.067383] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8665.071236] LustreError: 370567:0:(osd_handler.c:8546:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 8665.081664] LustreError: 370567:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 8665.085254] LustreError: 370567:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 8665.089239] LustreError: 370567:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 8665.092855] LustreError: 370567:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 8668.391501] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8668.395471] LustreError: 370964:0:(osd_handler.c:8546:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 8668.408318] LustreError: 370964:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 8668.411588] LustreError: 370964:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 8668.415541] LustreError: 370964:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 8668.419961] LustreError: 370964:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 8676.728318] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 8678.017405] Key type lgssc unregistered [ 8678.168743] LNet: 371964:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8678.172812] LNetError: 371964:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8678.181456] LNet: Removed LNI 192.168.204.107@tcp [ 8678.563945] Key type .llcrypt unregistered [ 8678.566713] Key type ._llcrypt unregistered [ 8686.858526] Key type ._llcrypt registered [ 8686.860242] Key type .llcrypt registered [ 8686.914508] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [ 8694.554250] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 8695.066880] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8695.085310] alg: No test for adler32 (adler32-zlib) [ 8695.989670] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 8696.117854] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 8697.719556] Key type lgssc registered [ 8698.220687] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8714.624494] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8714.638321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8715.758491] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8715.779586] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8715.824711] Lustre: lustre-MDT0000: new disk, initializing [ 8715.861284] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8715.877590] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8717.893872] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8723.973144] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8724.031949] Lustre: 375091:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8724.069075] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8724.072358] Lustre: Skipped 1 previous similar message [ 8724.111397] Lustre: lustre-MDT0001: new disk, initializing [ 8724.164385] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8724.183501] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8724.193910] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8726.146996] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8730.950691] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8733.612556] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8736.727673] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8736.849872] Lustre: lustre-OST0000: new disk, initializing [ 8736.851853] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8736.855986] Lustre: 377057:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8736.901046] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8737.251894] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8737.257671] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8737.300035] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8739.496273] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8744.556152] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8746.661825] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8746.757734] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8748.679058] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8748.784943] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8752.611463] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8752.611970] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8752.613264] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8752.615495] LustreError: Skipped 1 previous similar message [ 8752.621305] Lustre: Skipped 1 previous similar message [ 8755.829835] Lustre: server umount lustre-OST0000 complete [ 8757.733538] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8757.735497] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8757.741477] Lustre: Skipped 1 previous similar message [ 8757.747518] Lustre: Skipped 2 previous similar messages [ 8762.848625] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8762.851571] Lustre: Skipped 1 previous similar message [ 8763.770897] Lustre: server umount lustre-MDT0000 complete [ 8765.360478] LustreError: 375883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789526725 with bad export cookie 1139855731073998595 [ 8765.363977] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8765.367525] LustreError: 375883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8765.502644] Lustre: server umount lustre-MDT0001 complete [ 8769.099660] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 8769.901445] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 22:45:28 (1789526728) [ 8773.159312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8773.461847] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8775.463393] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8778.720336] LustreError: 379516:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8778.728608] LustreError: 379516:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8779.997346] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8780.179547] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8782.382656] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8786.062952] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8788.950255] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8792.316121] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8792.481528] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8795.035491] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8798.702568] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8802.002496] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8802.095982] Lustre: lustre-OST0001: new disk, initializing [ 8802.099940] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8802.105771] Lustre: 382347:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8805.214412] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8806.049580] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8806.057637] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8806.104283] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8810.580215] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8816.614981] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8816.619863] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8816.628482] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8816.637291] Lustre: Skipped 1 previous similar message [ 8818.974792] Lustre: server umount lustre-OST0001 complete [ 8821.733421] LustreError: 382787:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8821.744164] LustreError: 382787:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8826.854621] LustreError: 381332:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8826.868555] LustreError: 381332:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8831.968918] LustreError: 381335:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8831.970236] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8831.976156] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8831.976165] Lustre: Skipped 1 previous similar message [ 8831.985243] LustreError: 381335:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8831.994175] Lustre: Skipped 2 previous similar messages [ 8837.093356] LustreError: 381335:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8837.100646] LustreError: 381335:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8842.216046] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8842.223031] Lustre: Skipped 4 previous similar messages [ 8846.303449] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8846.389551] Lustre: server umount lustre-OST0000 complete [ 8851.938516] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8851.945666] LustreError: Skipped 1 previous similar message [ 8851.948958] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8854.720460] Lustre: server umount lustre-MDT0000 complete [ 8856.536288] LustreError: 379497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789526816 with bad export cookie 1139855731073999484 [ 8856.538519] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8856.545847] LustreError: 379497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8862.762647] Lustre: server umount lustre-MDT0001 complete [ 8867.698811] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 8869.072135] Key type lgssc unregistered [ 8869.232634] LNet: 384333:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8869.237953] LNetError: 384333:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8870.311160] LNet: Removed LNI 192.168.204.107@tcp [ 8870.740619] Key type .llcrypt unregistered [ 8870.742027] Key type ._llcrypt unregistered [ 8893.590703] Key type ._llcrypt registered [ 8893.592066] Key type .llcrypt registered [ 8893.635308] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 8894.220153] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8894.230214] alg: No test for adler32 (adler32-zlib) [ 8895.105018] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 8895.214922] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 8896.807165] Key type lgssc registered [ 8897.246522] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8900.346146] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8900.356191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8901.493815] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8901.505266] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8901.579991] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8903.341170] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8907.185826] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8907.247717] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8907.261923] Lustre: 387100:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8907.389837] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8909.265293] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8912.588241] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8914.922436] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8917.985422] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8917.990166] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8917.999841] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8920.544916] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8920.548472] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8925.664568] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8925.667378] Lustre: Skipped 2 previous similar messages [ 8930.271131] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8930.359951] Lustre: server umount lustre-MDT0000 complete [ 8932.088551] LustreError: 387095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789526891 with bad export cookie 8727610986806817338 [ 8932.094378] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8932.097538] LustreError: 387095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8932.246805] Lustre: server umount lustre-MDT0001 complete [ 8936.846716] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 22:48:15 (1789526895) [ 8939.908449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8940.115189] LustreError: 389616:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8940.122480] LustreError: 389616:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 8940.137910] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8942.038305] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8945.633783] LustreError: 389617:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8946.420773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8946.621228] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8948.539090] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8952.081803] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8954.850383] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8957.921254] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8958.021067] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8958.136042] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8960.871978] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8964.237885] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8973.959203] Lustre: Permanently deactivating lustre-OST0000 [ 8973.966851] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 8983.545097] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 8984.791255] LustreError: 392745:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 8984.797379] LustreError: 392745:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [ 8984.801576] LustreError: 392745:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 8985.712488] LustreError: 392793:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [ 8985.717990] LustreError: 392793:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 8992.804617] Lustre: server umount lustre-OST0000 complete [ 8994.119806] LustreError: 393042:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [ 8994.125958] LustreError: 393042:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 8995.808384] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8995.810070] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8995.815080] Lustre: Skipped 1 previous similar message [ 8995.820510] Lustre: Skipped 1 previous similar message [ 9000.930179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9000.934815] Lustre: Skipped 1 previous similar message [ 9005.064983] Lustre: server umount lustre-MDT0000 complete [ 9006.048162] LustreError: 389617:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9006.055878] LustreError: 389617:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9006.832230] LustreError: 389596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789526966 with bad export cookie 8727610986806817905 [ 9006.835155] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9006.838074] LustreError: 389596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9016.359107] Lustre: server umount lustre-MDT0001 complete [ 9020.624350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9022.732837] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 9024.139546] LustreError: 394198:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 9027.271855] Lustre: 394538:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.107@tcp [ 9028.227920] Lustre: server umount MGS complete [ 9033.158109] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9033.366371] LustreError: 395262:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9033.373458] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 9033.378906] Lustre: Skipped 1 previous similar message [ 9033.402657] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9035.111272] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 9038.718820] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9038.815855] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 9038.839468] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 9040.567658] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 9043.738941] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9046.152362] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9046.943389] Lustre: Permanently reactivating lustre-OST0000 [ 9046.951091] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 9046.954600] Lustre: Skipped 2 previous similar messages [ 9073.120575] LustreError: 397047:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 9083.881455] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9084.852414] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9084.880361] Lustre: Skipped 1 previous similar message [ 9094.415540] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 9105.357838] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9113.574821] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9113.581569] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9113.612369] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9114.603745] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9126.367628] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9126.571614] Lustre: server umount lustre-OST0000 complete [ 9130.976597] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9130.981278] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9130.987259] Lustre: Skipped 1 previous similar message [ 9136.768605] Lustre: server umount lustre-MDT0000 complete [ 9140.001647] LustreError: 396834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789527099 with bad export cookie 8727610986806818941 [ 9140.014234] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9140.019474] LustreError: 396834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9140.261976] Lustre: server umount lustre-MDT0001 complete [ 9148.585705] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 9150.558800] Key type lgssc unregistered [ 9150.789869] LNet: 399815:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9150.798049] LNetError: 399815:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9150.816450] LNet: Removed LNI 192.168.204.107@tcp [ 9151.402132] Key type .llcrypt unregistered [ 9151.406314] Key type ._llcrypt unregistered [ 9164.661507] Key type ._llcrypt registered [ 9164.664658] Key type .llcrypt registered [ 9164.737935] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [ 9176.434352] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 9177.006724] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9177.048675] alg: No test for adler32 (adler32-zlib) [ 9178.005315] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 9178.145386] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 9179.791419] Key type lgssc registered [ 9180.509760] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9206.663078] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9206.706505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9207.877605] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9207.903610] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9207.972308] Lustre: lustre-MDT0000: new disk, initializing [ 9208.068783] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9208.082558] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9211.527484] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 9220.027861] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9220.111936] Lustre: 402945:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9220.153099] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9220.159823] Lustre: Skipped 1 previous similar message [ 9220.224473] Lustre: lustre-MDT0001: new disk, initializing [ 9220.283473] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9220.333194] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9220.339949] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9223.962061] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 9231.413597] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9235.760733] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9241.236743] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9241.446437] Lustre: lustre-OST0000: new disk, initializing [ 9241.454073] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9241.460290] Lustre: 404916:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9241.524645] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9245.716509] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 9249.827285] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9249.847643] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9249.926705] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9252.626842] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9255.859964] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9255.999902] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9259.375450] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9259.573543] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9265.120110] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9265.125625] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9265.138768] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9265.635576] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9265.637383] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9265.660580] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9270.757722] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9270.764881] Lustre: Skipped 1 previous similar message [ 9275.359210] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9275.492659] Lustre: server umount lustre-OST0000 complete [ 9280.992906] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9281.011810] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9281.020294] Lustre: Skipped 1 previous similar message [ 9284.732221] Lustre: server umount lustre-MDT0000 complete [ 9287.391308] LustreError: 402938:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789527247 with bad export cookie 7087656185835465601 [ 9287.398725] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9287.627824] Lustre: server umount lustre-MDT0001 complete [ 9293.708038] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 9294.978536] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 22:54:12 (1789527252) [ 9300.201306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9300.551054] LustreError: 407374:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9300.576217] LustreError: 407374:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9300.625081] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9303.805703] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 9306.087925] LustreError: 407375:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9310.181363] LustreError: 407374:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9310.588463] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9310.886743] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9314.470765] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 9320.739368] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9325.465972] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9331.049252] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9331.326979] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9336.138755] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 9342.268407] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9343.884684] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 9350.453780] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 9356.257972] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9356.259808] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9356.273512] Lustre: Skipped 1 previous similar message [ 9356.276029] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9356.299471] LustreError: Skipped 1 previous similar message [ 9360.309123] Lustre: server umount lustre-OST0000 complete [ 9366.500502] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9366.524594] Lustre: Skipped 1 previous similar message [ 9366.533908] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9366.538604] Lustre: Skipped 1 previous similar message [ 9369.835926] Lustre: server umount lustre-MDT0000 complete [ 9371.625218] LustreError: 407375:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9371.643976] LustreError: 407375:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9372.812240] LustreError: 407356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789527332 with bad export cookie 7087656185835466483 [ 9372.813497] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9372.822551] LustreError: 407356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 9373.096559] Lustre: server umount lustre-MDT0001 complete [ 9382.319174] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 9384.527855] Key type lgssc unregistered [ 9384.832930] LNet: 411096:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9384.846704] LNetError: 411096:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9385.896521] LNet: Removed LNI 192.168.204.107@tcp [ 9386.633846] Key type .llcrypt unregistered [ 9386.635743] Key type ._llcrypt unregistered [ 9406.565499] Key type ._llcrypt registered [ 9406.569063] Key type .llcrypt registered [ 9406.694398] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 9406.963561] Key type .llcrypt unregistered [ 9406.967269] Key type ._llcrypt unregistered [ 9423.153843] Key type ._llcrypt registered [ 9423.156535] Key type .llcrypt registered [ 9423.259903] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 9424.068838] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9424.091713] alg: No test for adler32 (adler32-zlib) [ 9425.063872] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 9425.240952] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 9426.895595] Key type lgssc registered [ 9427.911747] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9437.053593] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9437.073713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9438.616682] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9444.557199] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 9448.930793] LustreError: 413299:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9448.949985] LustreError: 413299:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9454.054247] LustreError: 413300:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9457.831905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9458.652835] LustreError: 413299:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9458.678820] LustreError: 413299:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9458.709402] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9463.788543] LustreError: 413300:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9463.801848] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9463.809699] LustreError: 413300:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9466.438944] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 9468.901789] LustreError: 413299:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9468.947366] LustreError: 413299:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9475.905661] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9479.149175] LustreError: 413300:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9479.170773] LustreError: 413300:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9484.710763] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9485.175237] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9489.668936] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 9499.620517] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9499.630625] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9499.655417] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9500.647314] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9500.655236] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9500.669423] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9500.688319] Lustre: Skipped 1 previous similar message [ 9505.112685] Lustre: server umount lustre-OST0000 complete [ 9510.883826] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9510.886566] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9510.904138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9510.922825] Lustre: Skipped 1 previous similar message [ 9515.042250] Lustre: server umount lustre-MDT0000 complete [ 9516.001526] LustreError: 413296:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9516.021342] LustreError: 413296:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9518.346732] LustreError: 415495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789527478 with bad export cookie 249303003774759708 [ 9518.349644] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9518.356798] LustreError: 415495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9518.607957] Lustre: server umount lustre-MDT0001 complete [ 9529.891729] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 9532.560595] Key type lgssc unregistered [ 9532.878571] LNet: 416449:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9532.896776] LNetError: 416449:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9532.917354] LNet: Removed LNI 192.168.204.107@tcp [ 9533.695438] Key type .llcrypt unregistered [ 9533.701695] Key type ._llcrypt unregistered [ 9556.328182] Key type ._llcrypt registered [ 9556.333251] Key type .llcrypt registered [ 9556.466246] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 9557.497367] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9557.517912] alg: No test for adler32 (adler32-zlib) [ 9558.527455] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 9558.735341] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 9560.407224] Key type lgssc registered [ 9561.536209] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9569.169439] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9569.191031] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9584.799618] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 9589.919536] LustreError: 417837:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.107@tcp: failed processing log, type 4: rc = -110 [ 9616.607218] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9623.702988] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 9632.041753] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9639.119557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9639.611477] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9639.686477] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 9644.346540] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 9645.035097] LustreError: 417862:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9645.079529] LustreError: 417862:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9650.160440] LustreError: 418896:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9651.171951] LustreError: 418874:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9651.198263] LustreError: 418874:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9652.874888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9653.179289] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9657.427460] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 9664.689639] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9670.169914] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9678.360028] Lustre: server umount lustre-OST0000 complete [ 9683.939686] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9683.940911] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9683.949156] Lustre: Skipped 1 previous similar message [ 9683.968193] Lustre: Skipped 1 previous similar message [ 9689.058518] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9689.069990] Lustre: Skipped 1 previous similar message [ 9691.205347] Lustre: server umount lustre-MDT0000 complete [ 9694.175130] Lustre: 417174:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789527638/real 1789527638] req@00000000111e0ffe x1876455606745472/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789527654 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9694.176196] LustreError: 418875:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9694.196267] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9694.222957] LustreError: 418875:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9694.927290] LustreError: 417845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789527654 with bad export cookie 10131406738855979626 [ 9694.934408] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9694.940609] LustreError: 417845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9695.387042] Lustre: server umount lustre-MDT0001 complete [ 9707.300542] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 9710.256222] Key type lgssc unregistered [ 9710.540455] LNet: 421417:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9710.550895] LNetError: 421417:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9710.567053] LNet: Removed LNI 192.168.204.107@tcp [ 9711.347844] Key type .llcrypt unregistered [ 9711.350492] Key type ._llcrypt unregistered [ 9733.723817] Key type ._llcrypt registered [ 9733.732604] Key type .llcrypt registered [ 9733.839378] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 9734.752427] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9734.773185] alg: No test for adler32 (adler32-zlib) [ 9735.773633] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 9735.946456] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 9737.587650] Key type lgssc registered [ 9738.570431] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9745.751623] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9745.766908] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9747.168092] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9750.720905] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 9757.656396] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9757.673224] LustreError: 422844:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9757.698219] LustreError: 422844:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9757.976906] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9761.810946] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 9767.820354] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9772.867842] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9778.785251] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9779.080826] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9784.457546] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 9791.554373] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9792.454988] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 9802.326094] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.7@tcp (stopping) [ 9802.732079] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9802.747539] Lustre: Skipped 1 previous similar message [ 9805.885437] Lustre: server umount lustre-MDT0000 complete [ 9807.435806] LustreError: 422840:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9808.705951] LustreError: 422825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.7@tcp arrived at 1789527768 with bad export cookie 11366104244420244308 [ 9810.473307] LustreError: 422824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789527770 with bad export cookie 11366104244420243643 [ 9810.484687] LustreError: 422824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9810.495927] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9812.959990] LustreError: 422844:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9812.962192] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9812.998396] LustreError: 422844:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 9813.022551] Lustre: Skipped 1 previous similar message [ 9813.034159] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9813.036542] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9813.049849] Lustre: Skipped 3 previous similar messages [ 9815.969979] LustreError: 423922:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9815.972232] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9815.986859] LustreError: 423922:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9816.823245] Lustre: server umount lustre-OST0000 complete [ 9821.095933] LustreError: 422844:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9821.112251] LustreError: 422844:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9823.902953] Lustre: server umount lustre-MDT0001 complete [ 9834.149633] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 9837.039279] Key type lgssc unregistered [ 9837.289644] LNet: 426528:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9837.305370] LNetError: 426528:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9837.324484] LNet: Removed LNI 192.168.204.107@tcp [ 9838.103266] Key type .llcrypt unregistered [ 9838.105062] Key type ._llcrypt unregistered [ 9864.051332] Key type ._llcrypt registered [ 9864.053665] Key type .llcrypt registered [ 9864.138646] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 9864.950766] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9864.975267] alg: No test for adler32 (adler32-zlib) [ 9866.194793] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 9866.407743] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 9868.087232] Key type lgssc registered [ 9869.587960] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9877.126703] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9877.142208] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9878.486729] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9882.902340] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 9888.741555] LustreError: 427952:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9888.756359] LustreError: 427952:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9891.211973] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9891.677406] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9895.699656] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 9902.992829] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9908.901219] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9915.301116] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9915.603360] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9921.990984] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 9924.863287] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 9929.901746] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9935.333067] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9935.340558] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9935.353086] Lustre: Skipped 1 previous similar message [ 9937.376940] Lustre: server umount lustre-MDT0001 complete [ 9938.024185] LustreError: 427948:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9940.454782] LustreError: 427953:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9940.484062] LustreError: 427953:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9945.570085] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9945.576496] LustreError: 427952:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9945.577030] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9945.580061] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9945.624575] Lustre: Skipped 1 previous similar message [ 9950.153841] Lustre: server umount lustre-OST0000 complete [ 9950.692077] LustreError: 429005:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9954.074229] Lustre: server umount lustre-MDT0000 complete [ 9968.588946] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 9971.040734] Key type lgssc unregistered [ 9971.291504] LNet: 431636:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9971.298872] LNetError: 431636:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9971.314727] LNet: Removed LNI 192.168.204.107@tcp [ 9972.115390] Key type .llcrypt unregistered [ 9972.117242] Key type ._llcrypt unregistered [ 9994.940473] Key type ._llcrypt registered [ 9994.943533] Key type .llcrypt registered [ 9995.047499] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [10007.301945] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [10008.146023] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10008.169350] alg: No test for adler32 (adler32-zlib) [10009.194521] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [10009.416280] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [10011.055226] Key type lgssc registered [10012.023038] Lustre: Echo OBD driver; http://www.lustre.org/ [10039.053919] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [10049.396848] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10049.421579] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10050.574339] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10050.604638] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10050.654816] Lustre: lustre-MDT0000: new disk, initializing [10050.708720] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10050.720196] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10053.556818] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [10063.339795] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10063.500392] Lustre: lustre-OST0000: new disk, initializing [10063.503242] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10063.506917] Lustre: Skipped 1 previous similar message [10063.509666] Lustre: 436645:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10063.560400] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10065.890936] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [10065.899260] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [10065.922140] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [10067.075830] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [10073.587846] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10084.629270] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10084.713658] Lustre: 435623:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10084.785937] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10084.862288] Lustre: lustre-MDT0001: new disk, initializing [10084.962969] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10085.008057] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [10085.019819] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [10088.061548] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [10094.511767] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [10094.670259] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [10097.506128] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [10097.648522] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [10101.220365] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [10101.220640] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10101.236309] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10101.240730] Lustre: Skipped 1 previous similar message [10106.340958] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10106.344439] Lustre: Skipped 2 previous similar messages [10106.766256] Lustre: server umount lustre-MDT0001 complete [10116.586231] LustreError: 435632:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10116.601689] LustreError: 435632:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [10121.697237] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10121.700858] LustreError: 435637:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10121.704834] Lustre: Skipped 1 previous similar message [10121.726111] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10121.732774] Lustre: Skipped 1 previous similar message [10126.821052] LustreError: 435638:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10126.821074] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10127.486812] Lustre: server umount lustre-OST0000 complete [10144.223215] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10144.388089] Lustre: server umount lustre-MDT0000 complete [10152.144769] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [10154.001954] Key type lgssc unregistered [10154.215745] LNet: 440393:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10154.222836] LNetError: 440393:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10154.235780] LNet: Removed LNI 192.168.204.107@tcp [10154.814141] Key type .llcrypt unregistered [10154.816799] Key type ._llcrypt unregistered [10178.756554] Key type ._llcrypt registered [10178.759305] Key type .llcrypt registered [10178.830344] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [10179.005358] Key type .llcrypt unregistered [10179.008599] Key type ._llcrypt unregistered [10187.911582] Key type ._llcrypt registered [10187.913283] Key type .llcrypt registered [10187.974545] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [10188.426555] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10188.470715] alg: No test for adler32 (adler32-zlib) [10189.410481] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [10189.555099] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [10191.167239] Key type lgssc registered [10191.736554] Lustre: Echo OBD driver; http://www.lustre.org/ [10207.640641] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10207.654876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10208.805250] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10208.823757] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10208.872679] Lustre: lustre-MDT0000: new disk, initializing [10208.921183] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10208.933277] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10211.186990] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [10217.183702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10217.240291] Lustre: 443736:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10217.272075] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10217.275934] Lustre: Skipped 1 previous similar message [10217.316777] Lustre: lustre-MDT0001: new disk, initializing [10217.378365] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10217.398528] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10217.406652] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10219.323442] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [10224.350740] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10227.022635] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10230.663393] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10230.812223] Lustre: lustre-OST0000: new disk, initializing [10230.815574] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10230.819761] Lustre: 445701:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10230.863683] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10232.391499] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10232.396509] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10232.464241] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10233.746873] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [10238.902844] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10243.043000] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10243.043327] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10243.048783] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10243.053168] Lustre: Skipped 1 previous similar message [10243.057396] Lustre: Skipped 1 previous similar message [10248.122815] Lustre: server umount lustre-MDT0000 complete [10249.826639] LustreError: 443728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789528209 with bad export cookie 9965097484533380094 [10249.830627] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10249.831621] LustreError: 443728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10249.973555] Lustre: server umount lustre-MDT0001 complete [10261.951197] Lustre: server umount lustre-OST0000 complete [10266.756324] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 23:10:24 (1789528224) [10270.537322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10270.774607] LustreError: 447727:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10270.785353] LustreError: 447727:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [10270.819606] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10272.979981] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [10276.321389] LustreError: 447728:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10277.677203] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10277.860735] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10279.971758] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [10284.055172] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10287.099845] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10303.785679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10303.927309] Lustre: Found index 0 for lustre-OST0000, updating log [10304.038397] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10307.216174] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [10311.223426] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10316.338915] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [10321.377995] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10321.378597] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10321.389224] Lustre: Skipped 1 previous similar message [10321.395085] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10322.403324] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10324.452736] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10324.459697] Lustre: Skipped 1 previous similar message [10324.989260] Lustre: server umount lustre-OST0000 complete [10339.298292] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10339.302210] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10339.310390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10344.502780] Lustre: server umount lustre-MDT0000 complete [10345.952837] LustreError: 448318:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10345.960801] LustreError: 448318:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10346.477353] LustreError: 447707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789528306 with bad export cookie 9965097484533381487 [10346.480383] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10346.483959] LustreError: 447707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10346.630589] Lustre: server umount lustre-MDT0001 complete [10350.742090] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 23:11:48 (1789528308) [10363.615441] LNet: 1 local NIs in recovery (showing 1): 192.168.204.107@tcp [10933.776504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10933.913570] LustreError: 454128:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10933.932741] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10935.196420] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [10937.890676] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10938.017307] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10939.388693] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [10941.835993] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10943.622624] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10958.754485] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10958.828829] Lustre: Found index 0 for lustre-OST0000, updating log [10958.929771] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10960.723606] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [10960.998642] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [10962.951883] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10964.577731] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10964.581016] Lustre: Skipped 1 previous similar message [10964.582409] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10964.583804] Lustre: Skipped 2 previous similar messages [10965.600698] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10965.602336] Lustre: Skipped 1 previous similar message [10969.673978] Lustre: server umount lustre-OST0000 complete [10971.104373] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10971.105676] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10971.109580] Lustre: Skipped 1 previous similar message [10975.200362] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10975.202330] Lustre: Skipped 8 previous similar messages [10977.175619] Lustre: server umount lustre-MDT0000 complete [10977.248120] LustreError: 454128:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10977.252989] LustreError: 454128:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10978.396260] LustreError: 456023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789528938 with bad export cookie 9965097484533382558 [10978.398855] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10978.400113] LustreError: 456023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10978.506355] Lustre: server umount lustre-MDT0001 complete [10990.303279] LNet: 1 local NIs in recovery (showing 1): 192.168.204.107@tcp [10995.856632] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [10999.153874] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [11010.707633] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 23:22:49 (1789528969) [11016.105189] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [11017.021894] Key type lgssc unregistered [11021.215398] LNet: 460771:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11021.219277] LNetError: 460771:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11089.829318] LNet: Removed LNI 192.168.204.107@tcp [11090.169569] Key type .llcrypt unregistered [11090.170747] Key type ._llcrypt unregistered [11096.877984] Key type ._llcrypt registered [11096.879930] Key type .llcrypt registered [11096.917146] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [11097.220168] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11097.227930] alg: No test for adler32 (adler32-zlib) [11098.082285] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [11098.183749] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [11099.767267] Key type lgssc registered [11100.161771] Lustre: Echo OBD driver; http://www.lustre.org/ [11101.701739] LNet: Added LNI 192.168.204.107@tcp100 [8/256/0/180] [11102.015381] LNet: Added LNI 192.168.204.107@tcp101 [8/256/0/180] [11102.327587] LNet: Added LNI 192.168.204.107@tcp102 [8/256/0/180] [11102.640175] LNet: Added LNI 192.168.204.107@tcp103 [8/256/0/180] [11102.950636] LNet: Added LNI 192.168.204.107@tcp104 [8/256/0/180] [11103.249777] LNet: Added LNI 192.168.204.107@tcp105 [8/256/0/180] [11103.558880] LNet: Added LNI 192.168.204.107@tcp106 [8/256/0/180] [11103.874129] LNet: Added LNI 192.168.204.107@tcp107 [8/256/0/180] [11104.179600] LNet: Added LNI 192.168.204.107@tcp108 [8/256/0/180] [11104.496986] LNet: Added LNI 192.168.204.107@tcp109 [8/256/0/180] [11104.808602] LNet: Added LNI 192.168.204.107@tcp110 [8/256/0/180] [11105.113304] LNet: Added LNI 192.168.204.107@tcp111 [8/256/0/180] [11105.429456] LNet: Added LNI 192.168.204.107@tcp112 [8/256/0/180] [11105.726594] LNet: Added LNI 192.168.204.107@tcp113 [8/256/0/180] [11106.054417] LNet: Added LNI 192.168.204.107@tcp114 [8/256/0/180] [11106.392747] LNet: Added LNI 192.168.204.107@tcp115 [8/256/0/180] [11106.728066] LNet: Added LNI 192.168.204.107@tcp116 [8/256/0/180] [11107.062978] LNet: Added LNI 192.168.204.107@tcp117 [8/256/0/180] [11107.442520] LNet: Added LNI 192.168.204.107@tcp118 [8/256/0/180] [11107.766723] LNet: Added LNI 192.168.204.107@tcp119 [8/256/0/180] [11108.074842] LNet: Added LNI 192.168.204.107@tcp120 [8/256/0/180] [11108.394444] LNet: Added LNI 192.168.204.107@tcp121 [8/256/0/180] [11108.701684] LNet: Added LNI 192.168.204.107@tcp122 [8/256/0/180] [11109.036816] LNet: Added LNI 192.168.204.107@tcp123 [8/256/0/180] [11109.371836] LNet: Added LNI 192.168.204.107@tcp124 [8/256/0/180] [11109.681973] LNet: Added LNI 192.168.204.107@tcp125 [8/256/0/180] [11110.030098] LNet: Added LNI 192.168.204.107@tcp126 [8/256/0/180] [11110.402815] LNet: Added LNI 192.168.204.107@tcp127 [8/256/0/180] [11110.736474] LNet: Added LNI 192.168.204.107@tcp128 [8/256/0/180] [11111.100919] LNet: Added LNI 192.168.204.107@tcp129 [8/256/0/180] [11111.560526] LNet: Added LNI 192.168.204.107@tcp130 [8/256/0/180] [11112.081836] LNet: Added LNI 192.168.204.107@tcp131 [8/256/0/180] [11112.807691] LNet: Added LNI 192.168.204.107@tcp132 [8/256/0/180] [11113.580853] LNet: Added LNI 192.168.204.107@tcp133 [8/256/0/180] [11114.321383] LNet: Added LNI 192.168.204.107@tcp134 [8/256/0/180] [11114.997438] LNet: Added LNI 192.168.204.107@tcp135 [8/256/0/180] [11120.259489] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11120.279621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11121.409625] Lustre: 463766:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.107@tcp: can't add NID 192.168.204.107@tcp130: rc = -75 [11121.519845] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11121.540041] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11121.606666] Lustre: lustre-MDT0000: new disk, initializing [11121.667713] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11121.681675] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11125.365414] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [11138.122282] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11138.300663] Lustre: 463784:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11138.428810] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11138.436462] Lustre: Skipped 1 previous similar message [11138.547752] Lustre: lustre-MDT0001: new disk, initializing [11138.604858] Lustre: 464583:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.107@tcp: can't add NID 192.168.204.107@tcp130: rc = -75 [11138.625327] Lustre: 464583:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [11138.871328] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11138.919859] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11138.934292] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11143.857189] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [11156.135668] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11166.017663] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11174.242293] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11174.501417] Lustre: lustre-OST0000: new disk, initializing [11174.505260] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11174.510265] Lustre: 465757:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11174.577325] Lustre: 465762:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.107@tcp: can't add NID 192.168.204.107@tcp130: rc = -75 [11174.582246] Lustre: 465762:0:(lustre_peer.c:124:class_add_uuid()) Skipped 17 previous similar messages [11174.651290] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11180.054546] Lustre: 466027:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.107@tcp: can't add NID 192.168.204.107@tcp130: rc = -75 [11180.064974] Lustre: 466027:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [11180.134564] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11180.140029] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11180.278502] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11181.448906] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [11188.175246] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11195.874747] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11195.876800] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11195.876940] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11195.882178] Lustre: Skipped 1 previous similar message [11198.502349] Lustre: server umount lustre-OST0000 complete [11200.480387] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11200.484498] LustreError: Skipped 1 previous similar message [11200.486886] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11200.494193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11200.497978] Lustre: Skipped 1 previous similar message [11206.112692] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11206.115954] Lustre: Skipped 2 previous similar messages [11206.606813] Lustre: server umount lustre-MDT0000 complete [11208.403344] LustreError: 466911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789529168 with bad export cookie 14905122983745752303 [11208.404682] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11208.409678] LustreError: 466911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11208.568552] Lustre: server umount lustre-MDT0001 complete [11214.096339] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [11215.318879] Key type lgssc unregistered [11215.532666] LNet: 467669:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11215.537366] LNetError: 467669:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11215.543577] LNet: Removed LNI 192.168.204.107@tcp [11216.612084] LNet: Removed LNI 192.168.204.107@tcp100 [11216.614397] LNetError: 467669:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11216.619846] LNetError: 467669:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [11216.624907] LNet: Removed LNI 192.168.204.107@tcp101 [11217.631198] LNet: Removed LNI 192.168.204.107@tcp102 [11217.634634] LNetError: 467669:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11217.639594] LNetError: 467669:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [11217.645562] LNet: Removed LNI 192.168.204.107@tcp103 [11218.655426] LNet: Removed LNI 192.168.204.107@tcp104 [11219.679222] LNet: Removed LNI 192.168.204.107@tcp105 [11219.683098] LNetError: 467669:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11219.688508] LNetError: 467669:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [11219.693270] LNet: Removed LNI 192.168.204.107@tcp106 [11220.703195] LNet: Removed LNI 192.168.204.107@tcp107 [11221.727320] LNet: Removed LNI 192.168.204.107@tcp108 [11222.751550] LNet: Removed LNI 192.168.204.107@tcp109 [11223.775194] LNet: Removed LNI 192.168.204.107@tcp110 [11223.778846] LNetError: 467669:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11223.785699] LNetError: 467669:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [11223.791868] LNet: Removed LNI 192.168.204.107@tcp111 [11223.793642] LNet: Removed LNI 192.168.204.107@tcp112 [11223.799307] LNet: Removed LNI 192.168.204.107@tcp113 [11223.802961] LNet: Removed LNI 192.168.204.107@tcp114 [11223.806538] LNet: Removed LNI 192.168.204.107@tcp115 [11223.813734] LNet: Removed LNI 192.168.204.107@tcp116 [11223.818110] LNet: Removed LNI 192.168.204.107@tcp117 [11223.821672] LNet: Removed LNI 192.168.204.107@tcp118 [11223.825186] LNet: Removed LNI 192.168.204.107@tcp119 [11223.828261] LNet: Removed LNI 192.168.204.107@tcp120 [11223.831336] LNet: Removed LNI 192.168.204.107@tcp121 [11223.833273] LNet: Removed LNI 192.168.204.107@tcp122 [11223.836153] LNet: Removed LNI 192.168.204.107@tcp123 [11223.839141] LNet: Removed LNI 192.168.204.107@tcp124 [11223.843745] LNet: Removed LNI 192.168.204.107@tcp125 [11223.846746] LNet: Removed LNI 192.168.204.107@tcp126 [11223.850715] LNet: Removed LNI 192.168.204.107@tcp127 [11223.853580] LNet: Removed LNI 192.168.204.107@tcp128 [11223.857174] LNet: Removed LNI 192.168.204.107@tcp129 [11223.860103] LNet: Removed LNI 192.168.204.107@tcp130 [11223.862469] LNet: Removed LNI 192.168.204.107@tcp131 [11223.864710] LNet: Removed LNI 192.168.204.107@tcp132 [11223.866796] LNet: Removed LNI 192.168.204.107@tcp133 [11223.869613] LNet: Removed LNI 192.168.204.107@tcp134 [11223.879751] LNet: Removed LNI 192.168.204.107@tcp135 [11224.287296] Key type .llcrypt unregistered [11224.288573] Key type ._llcrypt unregistered [11237.149829] Key type ._llcrypt registered [11237.152203] Key type .llcrypt registered [11237.217443] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [11247.348452] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [11247.961199] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11247.994730] alg: No test for adler32 (adler32-zlib) [11248.954631] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [11249.146723] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [11250.792844] Key type lgssc registered [11251.458371] Lustre: Echo OBD driver; http://www.lustre.org/ [11270.366454] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 23:27:08 (1789529228) [11279.416519] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [11281.043877] Key type lgssc unregistered [11281.236700] LNet: 471722:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11281.241312] LNetError: 471722:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11281.254529] LNet: Removed LNI 192.168.204.107@tcp [11281.663484] Key type .llcrypt unregistered [11281.666130] Key type ._llcrypt unregistered [11291.953717] Key type ._llcrypt registered [11291.955436] Key type .llcrypt registered [11292.016382] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [11292.457148] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11292.470411] alg: No test for adler32 (adler32-zlib) [11293.374674] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [11293.541906] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [11295.175203] Key type lgssc registered [11295.871783] Lustre: Echo OBD driver; http://www.lustre.org/ [11301.249521] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11301.263066] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11302.440434] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11302.458459] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11302.516292] Lustre: lustre-MDT0000: new disk, initializing [11302.566361] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11302.583149] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11305.099713] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [11313.159152] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11313.255472] Lustre: 472907:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11313.329130] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11313.338978] Lustre: Skipped 1 previous similar message [11313.455236] Lustre: lustre-MDT0001: new disk, initializing [11313.563654] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11313.597649] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11313.606545] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11316.775988] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [11323.399876] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11327.359432] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11332.055783] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11332.237411] Lustre: lustre-OST0000: new disk, initializing [11332.240338] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11332.244977] Lustre: 474872:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11332.317685] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11336.301947] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [11341.346698] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11341.356222] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11341.437088] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11342.437692] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11344.848070] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing lnet_if_list [11348.365795] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [11348.388971] LNet: Added LNI 192.168.204.107@tcp10 [8/256/0/180] [11351.788842] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [11351.810645] LNet: Added LNI 192.168.204.107@tcp11 [8/256/0/180] [11354.404306] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [11354.425151] LNet: Added LNI 192.168.204.107@tcp12 [8/256/0/180] [11357.304722] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [11357.327600] LNet: Added LNI 192.168.204.107@tcp13 [8/256/0/180] [11359.869410] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [11359.899503] LNet: Added LNI 192.168.204.107@tcp14 [8/256/0/180] [11362.323206] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [11362.339936] LNet: Added LNI 192.168.204.107@tcp15 [8/256/0/180] [11364.874113] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [11364.890480] LNet: Added LNI 192.168.204.107@tcp16 [8/256/0/180] [11367.267317] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [11367.297949] LNet: Added LNI 192.168.204.107@tcp17 [8/256/0/180] [11369.930964] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [11369.957177] LNet: Added LNI 192.168.204.107@tcp18 [8/256/0/180] [11372.988378] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [11373.011537] LNet: Added LNI 192.168.204.107@tcp19 [8/256/0/180] [11375.946842] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [11375.971559] LNet: Added LNI 192.168.204.107@tcp20 [8/256/0/180] [11378.728413] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [11378.749953] LNet: Added LNI 192.168.204.107@tcp21 [8/256/0/180] [11381.564959] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [11381.584528] LNet: Added LNI 192.168.204.107@tcp22 [8/256/0/180] [11384.642041] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [11384.660301] LNet: Added LNI 192.168.204.107@tcp23 [8/256/0/180] [11387.938179] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [11387.956812] LNet: Added LNI 192.168.204.107@tcp24 [8/256/0/180] [11391.170736] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [11391.194208] LNet: Added LNI 192.168.204.107@tcp25 [8/256/0/180] [11394.723198] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [11394.743890] LNet: Added LNI 192.168.204.107@tcp26 [8/256/0/180] [11398.199694] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [11398.222598] LNet: Added LNI 192.168.204.107@tcp27 [8/256/0/180] [11401.735742] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [11401.759897] LNet: Added LNI 192.168.204.107@tcp28 [8/256/0/180] [11405.568763] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [11405.591218] LNet: Added LNI 192.168.204.107@tcp29 [8/256/0/180] [11409.778089] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [11409.801566] LNet: Added LNI 192.168.204.107@tcp30 [8/256/0/180] [11413.445841] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [11413.468314] LNet: Added LNI 192.168.204.107@tcp31 [8/256/0/180] [11417.345685] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [11417.372323] LNet: Added LNI 192.168.204.107@tcp32 [8/256/0/180] [11421.973465] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [11422.007143] LNet: Added LNI 192.168.204.107@tcp33 [8/256/0/180] [11426.238700] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [11426.256697] LNet: Added LNI 192.168.204.107@tcp34 [8/256/0/180] [11429.627521] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [11429.643960] LNet: Added LNI 192.168.204.107@tcp35 [8/256/0/180] [11433.838101] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [11433.854734] LNet: Added LNI 192.168.204.107@tcp36 [8/256/0/180] [11437.277291] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [11437.292964] LNet: Added LNI 192.168.204.107@tcp37 [8/256/0/180] [11440.334516] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [11440.356111] LNet: Added LNI 192.168.204.107@tcp38 [8/256/0/180] [11443.600481] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [11443.619130] LNet: Added LNI 192.168.204.107@tcp39 [8/256/0/180] [11446.589861] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [11446.604627] LNet: Added LNI 192.168.204.107@tcp40 [8/256/0/180] [11449.746789] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [11449.765989] LNet: Added LNI 192.168.204.107@tcp41 [8/256/0/180] [11453.061499] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [11453.081056] LNet: Added LNI 192.168.204.107@tcp42 [8/256/0/180] [11456.527060] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [11456.548820] LNet: Added LNI 192.168.204.107@tcp43 [8/256/0/180] [11460.171930] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [11460.189032] LNet: Added LNI 192.168.204.107@tcp44 [8/256/0/180] [11464.039832] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [11464.057026] LNet: Added LNI 192.168.204.107@tcp45 [8/256/0/180] [11468.069307] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [11468.086125] LNet: Added LNI 192.168.204.107@tcp46 [8/256/0/180] [11472.015899] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [11472.041170] LNet: Added LNI 192.168.204.107@tcp47 [8/256/0/180] [11476.070190] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [11476.085348] LNet: Added LNI 192.168.204.107@tcp48 [8/256/0/180] [11480.810415] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [11480.838406] LNet: Added LNI 192.168.204.107@tcp49 [8/256/0/180] [11485.586400] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [11485.611483] LNet: Added LNI 192.168.204.107@tcp50 [8/256/0/180] [11497.953612] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11497.960503] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11497.968699] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11500.001028] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11500.004059] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11500.026380] Lustre: Skipped 1 previous similar message [11500.030103] Lustre: Skipped 1 previous similar message [11502.783351] Lustre: server umount lustre-MDT0000 complete [11505.030343] LustreError: 472901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789529464 with bad export cookie 15663878136387833639 [11505.037470] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11505.038738] LustreError: 472901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11505.120882] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11505.127556] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11505.132172] Lustre: Skipped 1 previous similar message [11505.235821] Lustre: server umount lustre-MDT0001 complete [11515.807426] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [11520.341896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11529.695492] LustreError: 472179:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000004570c123 x1876457426108032/t0(0) o250->MGC192.168.204.107@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11529.840240] LustreError: 474867:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11529.850317] LustreError: 474867:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11529.870834] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11530.915330] LustreError: 474866:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11530.917361] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [11532.111340] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [11536.354555] LustreError: 474867:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11536.670773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11536.915657] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11539.250194] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [11543.726147] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11546.083910] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [11546.839710] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11560.054400] Lustre: server umount lustre-OST0000 complete [11562.464347] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11562.469680] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11562.477163] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11566.562871] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11566.564740] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11566.571747] Lustre: Skipped 1 previous similar message [11572.753079] Lustre: server umount lustre-MDT0000 complete [11574.919484] LustreError: 472899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789529534 with bad export cookie 15663878136387835935 [11574.921672] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11574.926161] LustreError: 472899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11584.556776] Lustre: server umount lustre-MDT0001 complete [11590.168648] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [11591.858383] Key type lgssc unregistered [11592.115780] LNet: 501848:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11592.123314] LNetError: 501848:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11592.128882] LNet: Removed LNI 192.168.204.107@tcp [11592.131594] LNet: Removed LNI 192.168.204.107@tcp10 [11592.134922] LNet: Removed LNI 192.168.204.107@tcp11 [11592.137130] LNet: Removed LNI 192.168.204.107@tcp12 [11592.139815] LNet: Removed LNI 192.168.204.107@tcp13 [11592.141832] LNet: Removed LNI 192.168.204.107@tcp14 [11592.144699] LNet: Removed LNI 192.168.204.107@tcp15 [11592.149616] LNet: Removed LNI 192.168.204.107@tcp16 [11592.151909] LNet: Removed LNI 192.168.204.107@tcp17 [11592.154440] LNet: Removed LNI 192.168.204.107@tcp18 [11592.156488] LNet: Removed LNI 192.168.204.107@tcp19 [11592.159154] LNet: Removed LNI 192.168.204.107@tcp20 [11592.161271] LNet: Removed LNI 192.168.204.107@tcp21 [11592.164393] LNet: Removed LNI 192.168.204.107@tcp22 [11592.167111] LNet: Removed LNI 192.168.204.107@tcp23 [11592.169464] LNet: Removed LNI 192.168.204.107@tcp24 [11592.171603] LNet: Removed LNI 192.168.204.107@tcp25 [11592.174210] LNet: Removed LNI 192.168.204.107@tcp26 [11592.176133] LNet: Removed LNI 192.168.204.107@tcp27 [11592.178287] LNet: Removed LNI 192.168.204.107@tcp28 [11592.180554] LNet: Removed LNI 192.168.204.107@tcp29 [11592.183920] LNet: Removed LNI 192.168.204.107@tcp30 [11592.186556] LNet: Removed LNI 192.168.204.107@tcp31 [11592.189286] LNet: Removed LNI 192.168.204.107@tcp32 [11592.192306] LNet: Removed LNI 192.168.204.107@tcp33 [11592.195480] LNet: Removed LNI 192.168.204.107@tcp34 [11592.200035] LNet: Removed LNI 192.168.204.107@tcp35 [11592.202876] LNet: Removed LNI 192.168.204.107@tcp36 [11592.205265] LNet: Removed LNI 192.168.204.107@tcp37 [11592.208133] LNet: Removed LNI 192.168.204.107@tcp38 [11592.210518] LNet: Removed LNI 192.168.204.107@tcp39 [11592.212971] LNet: Removed LNI 192.168.204.107@tcp40 [11592.215821] LNet: Removed LNI 192.168.204.107@tcp41 [11592.218401] LNet: Removed LNI 192.168.204.107@tcp42 [11592.221564] LNet: Removed LNI 192.168.204.107@tcp43 [11592.224424] LNet: Removed LNI 192.168.204.107@tcp44 [11592.227278] LNet: Removed LNI 192.168.204.107@tcp45 [11592.229467] LNet: Removed LNI 192.168.204.107@tcp46 [11592.231800] LNet: Removed LNI 192.168.204.107@tcp47 [11592.235463] LNet: Removed LNI 192.168.204.107@tcp48 [11592.239702] LNet: Removed LNI 192.168.204.107@tcp49 [11592.253723] LNet: Removed LNI 192.168.204.107@tcp50 [11592.684345] Key type .llcrypt unregistered [11592.686244] Key type ._llcrypt unregistered [11603.374227] Key type ._llcrypt registered [11603.376020] Key type .llcrypt registered [11603.447392] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [11611.506150] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [11611.931666] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11611.940800] alg: No test for adler32 (adler32-zlib) [11612.845137] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [11612.981418] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [11614.593297] Key type lgssc registered [11615.133673] Lustre: Echo OBD driver; http://www.lustre.org/ [11632.114654] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11632.127850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11633.271143] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11633.291353] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11633.354388] Lustre: lustre-MDT0000: new disk, initializing [11633.413137] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11633.427680] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11636.211864] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [11643.301640] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11643.395090] Lustre: 505016:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11643.445395] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11643.449406] Lustre: Skipped 1 previous similar message [11643.500284] Lustre: lustre-MDT0001: new disk, initializing [11643.588190] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11643.618689] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11643.623743] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11646.668075] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [11652.153795] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11654.871241] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11658.404528] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11658.517853] Lustre: *** cfs_fail_loc=90f, val=0*** [11658.576775] Lustre: lustre-OST0000: new disk, initializing [11658.579725] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11658.584465] Lustre: 507033:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11658.631107] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11661.669099] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [11666.975743] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11666.986824] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11667.027879] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11667.068531] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11672.034347] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11672.037463] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11672.039793] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11672.051124] Lustre: Skipped 1 previous similar message [11672.056275] Lustre: Skipped 1 previous similar message [11676.838618] Lustre: server umount lustre-OST0000 complete [11679.713677] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11679.718637] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11679.724899] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11682.274874] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11682.276783] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11682.282283] Lustre: Skipped 1 previous similar message [11684.715586] Lustre: server umount lustre-MDT0000 complete [11686.289430] LustreError: 505008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789529646 with bad export cookie 15071326249527152036 [11686.291576] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11686.294985] LustreError: 505008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11686.442742] Lustre: server umount lustre-MDT0001 complete [11690.702067] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [11691.954973] Key type lgssc unregistered [11692.121617] LNet: 508933:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11692.125455] LNetError: 508933:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11692.134485] LNet: Removed LNI 192.168.204.107@tcp [11692.537294] Key type .llcrypt unregistered [11692.539185] Key type ._llcrypt unregistered [11702.100258] Key type ._llcrypt registered [11702.101908] Key type .llcrypt registered [11702.155049] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [11702.598065] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11702.607627] alg: No test for adler32 (adler32-zlib) [11703.518363] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [11703.650802] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [11705.255498] Key type lgssc registered [11705.720374] Lustre: Echo OBD driver; http://www.lustre.org/ [11709.069334] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11709.078375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11710.250717] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11711.925579] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [11715.485996] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11715.628527] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11717.216332] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [11720.349551] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11722.723663] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11725.635700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11725.818476] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11728.482677] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [11731.908908] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11741.511363] loop0: detected capacity change from 0 to 204800000 [11741.579706] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11741.587817] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11741.593803] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11744.434096] loop0: detected capacity change from 0 to 204800000 [11744.572680] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11744.578188] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11744.583887] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11747.065512] loop0: detected capacity change from 0 to 204800000 [11749.674540] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11749.761879] Lustre: fs15246-OST0000: new disk, initializing [11749.765446] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [11749.770420] Lustre: 513783:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11749.807834] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11752.795876] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [11757.707276] loop1: detected capacity change from 0 to 204800000 [11760.124778] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11760.177077] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [11760.210177] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [11760.251915] Lustre: fs15246-MDT0000: new disk, initializing [11760.322523] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11760.334755] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11760.392167] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11761.441069] Lustre: 514905:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000041cf5785 x1876457855942912/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1789529737 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [11762.392788] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [11766.498047] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [11766.503352] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [11766.516987] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [11775.974900] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11775.980259] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [11778.207737] Lustre: server umount fs15246-MDT0000 complete [11781.092752] LustreError: 510183:0:(ldlm_lib.c:1199:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11781.103546] LustreError: 510183:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11782.126780] Lustre: server umount fs15246-OST0000 complete [11787.233283] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11787.237942] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11787.246457] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11788.256756] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11788.259350] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11788.266349] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11791.333106] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11791.336635] Lustre: Skipped 1 previous similar message [11792.990579] Lustre: server umount lustre-OST0000 complete [11796.448811] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11796.449437] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11796.454491] Lustre: Skipped 1 previous similar message [11800.874293] Lustre: server umount lustre-MDT0000 complete [11801.567858] LustreError: 510928:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11801.573833] LustreError: 510928:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11802.313797] LustreError: 510214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789529762 with bad export cookie 697608852656622244 [11802.315956] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11802.318867] LustreError: 510214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11802.460033] Lustre: server umount lustre-MDT0001 complete [11806.658150] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [11807.924350] Key type lgssc unregistered [11808.080951] LNet: 517996:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11808.085526] LNetError: 517996:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11808.096500] LNet: Removed LNI 192.168.204.107@tcp [11808.487270] Key type .llcrypt unregistered [11808.488943] Key type ._llcrypt unregistered [11818.215371] Key type ._llcrypt registered [11818.217083] Key type .llcrypt registered [11818.269166] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [11818.682408] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11818.761806] alg: No test for adler32 (adler32-zlib) [11819.685514] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [11819.807524] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [11821.423221] Key type lgssc registered [11822.029832] Lustre: Echo OBD driver; http://www.lustre.org/ [11839.865214] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [11844.362101] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [11861.439511] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11861.451224] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11862.552211] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11862.567323] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11862.606630] Lustre: lustre-MDT0000: new disk, initializing [11862.638411] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11862.646775] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11864.420965] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [11869.657588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11869.709261] Lustre: 522430:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11869.741415] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11869.743606] Lustre: Skipped 1 previous similar message [11869.784579] Lustre: lustre-MDT0001: new disk, initializing [11869.823588] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11869.839865] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11869.845494] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11871.494817] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [11875.798832] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11878.186423] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11881.253522] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11881.406330] Lustre: lustre-OST0000: new disk, initializing [11881.408798] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11881.412973] Lustre: 524396:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11881.455632] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11883.937974] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [11886.604797] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11886.611179] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11886.679911] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11888.674772] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11890.540925] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11890.634139] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11892.423607] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11892.522395] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11896.805691] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11896.806179] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11896.807543] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11896.810900] Lustre: Skipped 1 previous similar message [11899.501659] Lustre: server umount lustre-OST0000 complete [11901.922024] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11901.922758] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11901.927397] Lustre: Skipped 1 previous similar message [11901.933698] Lustre: Skipped 2 previous similar messages [11907.040618] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11907.044271] Lustre: Skipped 1 previous similar message [11907.494679] Lustre: server umount lustre-MDT0000 complete [11908.972438] LustreError: 522423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789529868 with bad export cookie 18065376920534821140 [11908.975640] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11908.977339] LustreError: 522423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11909.120154] Lustre: server umount lustre-MDT0001 complete [11918.622507] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [11923.235643] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [11937.065573] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 11628 sec ========= 23:38:15 (1789529895) [11937.783867] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 23:38:16 (1789529896) === [11939.281378] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 23:38:17 (1789529897) === [11957.132438] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [11958.469883] Key type lgssc unregistered [11958.639647] LNet: 531027:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11958.645158] LNetError: 531027:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11958.662929] LNet: Removed LNI 192.168.204.107@tcp [11959.044654] Key type .llcrypt unregistered [11959.046468] Key type ._llcrypt unregistered