[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 827856256 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002388] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004013] kvm-guest: setup PV IPIs [ 0.007392] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008033] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009018] pid_max: default: 32768 minimum: 301 [ 0.010134] LSM: Security Framework initializing [ 0.011054] Yama: becoming mindful. [ 0.012041] SELinux: Initializing. [ 0.014071] *** VALIDATE selinux *** [ 0.023189] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028477] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029165] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030126] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032055] *** VALIDATE tmpfs *** [ 0.033485] *** VALIDATE proc *** [ 0.034305] *** VALIDATE cgroup *** [ 0.035013] *** VALIDATE cgroup2 *** [ 0.037040] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038154] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040031] Spectre V2 : User space: Vulnerable [ 0.041009] Speculative Store Bypass: Vulnerable [ 0.044500] debug: unmapping init [mem 0xffffffff8e859000-0xffffffff8e860fff] [ 0.046219] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047736] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048025] ... version: 2 [ 0.049013] ... bit width: 48 [ 0.050008] ... generic registers: 4 [ 0.051011] ... value mask: 0000ffffffffffff [ 0.052015] ... max period: 00007fffffffffff [ 0.053011] ... fixed-purpose events: 3 [ 0.054012] ... event mask: 000000070000000f [ 0.056071] rcu: Hierarchical SRCU implementation. [ 0.058785] smp: Bringing up secondary CPUs ... [ 0.059670] x86: Booting SMP configuration: [ 0.060023] .... node #0, CPUs: #1 #2 #3 [ 0.070354] smp: Brought up 1 node, 4 CPUs [ 0.072021] smpboot: Max logical packages: 1 [ 0.073216] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.145367] node 0 deferred pages initialised in 69ms [ 0.147329] devtmpfs: initialized [ 0.148000] x86/mm: Memory block size: 128MB [ 0.148000] gcov: version magic: 0x41383552 [ 0.154342] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.158078] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.161303] pinctrl core: initialized pinctrl subsystem [ 0.164218] [ 0.165008] ************************************************************* [ 0.173019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.179011] ** ** [ 0.184026] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.189026] ** ** [ 0.194015] ** This means that this kernel is built to expose internal ** [ 0.199121] ** IOMMU data structures, which may compromise security on ** [ 0.201013] ** your system. ** [ 0.205138] ** ** [ 0.209018] ** If you see this message and you are not debugging the ** [ 0.213015] ** kernel, report this immediately to your vendor! ** [ 0.220012] ** ** [ 0.224013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.228013] ************************************************************* [ 0.239296] NET: Registered protocol family 16 [ 0.243867] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.248080] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.255080] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.261165] cpuidle: using governor menu [ 0.265765] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.268878] PCI: Using configuration type 1 for base access [ 0.272121] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.300240] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.307124] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.379000] cryptd: max_cpu_qlen set to 1000 [ 0.380503] ACPI: Added _OSI(Module Device) [ 0.382014] ACPI: Added _OSI(Processor Device) [ 0.386121] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.390016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.396026] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.415013] ACPI: Interpreter enabled [ 0.419616] ACPI: PM: (supports S0 S3 S4 S5) [ 0.420041] ACPI: Using IOAPIC for interrupt routing [ 0.425438] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.434621] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.449074] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.451036] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.454018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.457096] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.466361] acpiphp: Slot [2] registered [ 0.468127] acpiphp: Slot [5] registered [ 0.471207] acpiphp: Slot [6] registered [ 0.473155] acpiphp: Slot [7] registered [ 0.476145] acpiphp: Slot [8] registered [ 0.478160] acpiphp: Slot [9] registered [ 0.480238] acpiphp: Slot [10] registered [ 0.483144] acpiphp: Slot [3] registered [ 0.486930] acpiphp: Slot [4] registered [ 0.489142] acpiphp: Slot [11] registered [ 0.491133] acpiphp: Slot [12] registered [ 0.494332] acpiphp: Slot [13] registered [ 0.497154] acpiphp: Slot [14] registered [ 0.499159] acpiphp: Slot [15] registered [ 0.502124] acpiphp: Slot [16] registered [ 0.504560] acpiphp: Slot [17] registered [ 0.508152] acpiphp: Slot [18] registered [ 0.511463] acpiphp: Slot [19] registered [ 0.545185] acpiphp: Slot [20] registered [ 0.552940] acpiphp: Slot [21] registered [ 0.563668] acpiphp: Slot [22] registered [ 0.570408] acpiphp: Slot [23] registered [ 0.575358] acpiphp: Slot [24] registered [ 0.578123] acpiphp: Slot [25] registered [ 0.580242] acpiphp: Slot [26] registered [ 0.583219] acpiphp: Slot [27] registered [ 0.585099] acpiphp: Slot [28] registered [ 0.587098] acpiphp: Slot [29] registered [ 0.589106] acpiphp: Slot [30] registered [ 0.590063] acpiphp: Slot [31] registered [ 0.591052] PCI host bridge to bus 0000:00 [ 0.592016] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.595018] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.597016] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.600022] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.604031] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.608159] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.610764] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.614442] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.619220] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.635016] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.643014] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.646184] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.649018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.653022] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.656871] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.661136] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.665042] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.668185] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.676015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.803408] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.818015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.827881] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.838014] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.844015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.857016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.868013] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.924017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.933015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.947014] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.959339] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.968025] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.975016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.990000] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.990000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.993017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 1.065018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 1.119017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 1.132496] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 1.145094] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 1.153017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 1.172039] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 1.189432] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 1.196016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 1.201015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 1.214017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 1.228778] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 1.231470] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 1.234368] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 1.236324] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 1.239209] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 1.245035] iommu: Default domain type: Passthrough [ 1.257745] SCSI subsystem initialized [ 1.264697] ACPI: bus type USB registered [ 1.271286] usbcore: registered new interface driver usbfs [ 1.280580] usbcore: registered new interface driver hub [ 1.288122] usbcore: registered new device driver usb [ 1.296433] pps_core: LinuxPPS API ver. 1 registered [ 1.302015] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 1.316101] PTP clock support registered [ 1.324156] EDAC MC: Ver: 3.0.0 [ 1.326000] PCI: Using ACPI for IRQ routing [ 1.326000] NetLabel: Initializing [ 1.328018] NetLabel: domain hash size = 128 [ 1.331011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 1.333210] NetLabel: unlabeled traffic allowed by default [ 1.336192] vgaarb: loaded [ 1.338188] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.339010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.349980] clocksource: Switched to clocksource kvm-clock [ 1.575075] VFS: Disk quotas dquot_6.6.0 [ 1.576861] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.578968] *** VALIDATE ramfs *** [ 1.580083] *** VALIDATE hugetlbfs *** [ 1.581484] pnp: PnP ACPI init [ 1.583739] pnp: PnP ACPI: found 6 devices [ 1.602221] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.605728] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.608209] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.610726] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.614712] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.617418] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.619699] NET: Registered protocol family 2 [ 1.621811] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.627106] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.630915] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.636736] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.640240] TCP: Hash tables configured (established 65536 bind 65536) [ 1.643786] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.647142] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.650239] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.653687] NET: Registered protocol family 1 [ 1.656444] RPC: Registered named UNIX socket transport module. [ 1.659091] RPC: Registered udp transport module. [ 1.660603] RPC: Registered tcp transport module. [ 1.662536] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.665107] NET: Registered protocol family 44 [ 1.667324] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.669895] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.672092] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.674575] PCI: CLS 0 bytes, default 64 [ 1.676237] Unpacking initramfs... [ 2.006821] hrtimer: interrupt took 111810352 ns [ 5.193136] debug: unmapping init [mem 0xffff92a6bcc54000-0xffff92a6bffbffff] [ 5.197827] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 5.200286] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 5.204174] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 6.380503] Initialise system trusted keyrings [ 6.387247] Key type blacklist registered [ 6.393423] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 6.417439] zbud: loaded [ 6.423387] *** VALIDATE nfs *** [ 6.425772] *** VALIDATE nfs4 *** [ 6.428501] pstore: using deflate compression [ 6.440266] Platform Keyring initialized [ 6.857934] NET: Registered protocol family 38 [ 6.859715] Key type asymmetric registered [ 6.863616] Asymmetric key parser 'x509' registered [ 6.868790] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 6.877203] io scheduler mq-deadline registered [ 6.878857] io scheduler kyber registered [ 6.880556] io scheduler bfq registered [ 6.889280] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 6.891540] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 6.893851] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 6.896927] ACPI: Power Button [PWRF] [ 7.082963] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 7.350704] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 7.479636] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 7.485811] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 7.507725] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 7.535250] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 7.714739] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 7.744651] Non-volatile memory driver v1.3 [ 7.748402] Linux agpgart interface v0.103 [ 7.819533] virtio_blk virtio1: [vda] 150000 512-byte logical blocks (76.8 MB/73.2 MiB) [ 7.829199] vda: detected capacity change from 0 to 76800000 [ 7.854378] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 7.857850] vdb: detected capacity change from 0 to 1073741824 [ 7.876950] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 7.880806] vdc: detected capacity change from 0 to 2621440000 [ 7.929724] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 7.932868] vdd: detected capacity change from 0 to 2621440000 [ 7.950631] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 7.954150] vde: detected capacity change from 0 to 4294967296 [ 7.972732] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 7.976186] vdf: detected capacity change from 0 to 4294967296 [ 7.985580] libphy: Fixed MDIO Bus: probed [ 7.997710] usbcore: registered new interface driver usbserial_generic [ 7.999690] usbserial: USB Serial support registered for generic [ 8.002683] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 8.006967] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 8.008423] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 8.010359] mousedev: PS/2 mouse device common for all mice [ 8.012843] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 8.015428] rtc_cmos 00:05: RTC can wake from S4 [ 8.020284] rtc_cmos 00:05: registered as rtc0 [ 8.022071] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 8.024289] intel_pstate: CPU model not supported [ 8.027277] hid: raw HID events driver (C) Jiri Kosina [ 8.029236] usbcore: registered new interface driver usbhid [ 8.029627] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 8.030898] usbhid: USB HID core driver [ 8.036062] drop_monitor: Initializing network drop monitor service [ 8.038061] Initializing XFRM netlink socket [ 8.038111] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 8.039716] NET: Registered protocol family 10 [ 8.046047] Segment Routing with IPv6 [ 8.047191] NET: Registered protocol family 17 [ 8.050583] mpls_gso: MPLS GSO support [ 8.057880] RAS: Correctable Errors collector initialized. [ 8.061118] AVX version of gcm_enc/dec engaged. [ 8.062403] AES CTR mode by8 optimization enabled [ 8.891605] sched_clock: Marking stable (8891525101, 0)->(11363669202, -2472144101) [ 9.007102] registered taskstats version 1 [ 9.013168] Loading compiled-in X.509 certificates [ 9.044494] zswap: loaded using pool lzo/zbud [ 9.105224] Key type big_key registered [ 9.149406] Key type encrypted registered [ 9.151749] ima: No TPM chip found, activating TPM-bypass! [ 9.155218] ima: Allocated hash algorithm: sha1 [ 9.158155] ima: No architecture policies found [ 9.162687] evm: Initialising EVM extended attributes: [ 9.165378] evm: security.selinux [ 9.166960] evm: security.ima [ 9.206962] evm: security.capability [ 9.209388] evm: HMAC attrs: 0x1 [ 9.214601] rtc_cmos 00:05: setting system clock to 2026-09-17 08:49:45 UTC (1789634985) [ 9.227133] debug: unmapping init [mem 0xffffffff8f803000-0xffffffff8f9fffff] [ 9.231572] debug: unmapping init [mem 0xffffffff8e582000-0xffffffff8e858fff] [ 9.243233] Write protecting the kernel read-only data: 28672k [ 9.324210] debug: unmapping init [mem 0xffffffff8cc03000-0xffffffff8cdfffff] [ 9.330529] debug: unmapping init [mem 0xffffffff8d514000-0xffffffff8d5fffff] [ 9.897393] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 9.905479] systemd[1]: Detected virtualization kvm. [ 9.907176] systemd[1]: Detected architecture x86-64. [ 9.909521] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 10.013933] systemd[1]: No hostname configured. [ 10.015503] systemd[1]: Set hostname to . [ 10.017803] random: systemd: uninitialized urandom read (16 bytes read) [ 10.020056] systemd[1]: Initializing machine ID from random generator. [ 10.320809] random: systemd: uninitialized urandom read (16 bytes read) [ 10.325719] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 10.341947] random: systemd: uninitialized urandom read (16 bytes read) [ 10.346444] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 10.359201] systemd[1]: Started Memstrack Anylazing Service. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Slices. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Initrd Root Device. Starting Setup Virtual Console... [ OK ] Reached target Timers. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 11.376151] device-mapper: uevent: version 1.0.3 [ 11.379529] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted Kernel Configuration File System. [ 14.456842] virtio_net virtio0 ens2: renamed from eth0 [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 15.919106] random: fast init done [ 21.012263] scsi host0: ata_piix [ 21.052196] scsi host1: ata_piix [ 21.054967] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 21.059659] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 21.743610] random: crng init done [ 21.751345] random: 7 urandom warning(s) missed due to ratelimiting [ 24.787406] dracut-initqueue[581]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 27.257411] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 31.117615] printk: systemd: 26 output lines suppressed due to ratelimiting [ 32.007878] SELinux: Disabled at runtime. [ 32.154952] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 32.176951] systemd[1]: Detected virtualization kvm. [ 32.181176] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 34.782011] systemd[1]: initrd-switch-root.service: Succeeded. [ 34.793351] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 34.840639] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 34.870054] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 34.885630] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 34.935123] systemd[1]: Starting Journal Service... Starting Journal Service... [ 35.008248] systemd[1]: Listening on RPCbind Server Activation Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice User and Session Slice. Mounting Kernel Debug File System... Starting Remount Root and Kernel File Systems... [ OK ] Reached target Paths. Mounting POSIX Message Queue File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target Slices. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Huge Pages File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ 35.992517] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 37.115571] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 39.351766] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 39.546521] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 40.986153] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 41.493230] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (9s / no limit) [** ] A start job is running for Configur…-only root support (9s / no limit) [*** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (11s / no limit) [ ***] A start job is running for Configur…only root support (11s / no limit) [ **] A start job is running for Configur…only root support (12s / no limit) [ *] A start job is running for Configur…only root support (12s / no limit) [ **] A start job is running for Configur…only root support (13s / no limit) [ ***] A start job is running for Configur…only root support (13s / no limit) [ *** ] A start job is running for Configur…only root support (14s / no limit) [ *** ] A start job is running for Configur…only root support (14s / no limit) [*** ] A start job is running for Configur…only root support (15s / no limit) [** ] A start job is running for Configur…only root support (15s / no limit)[ 50.534846] Key type dns_resolver registered [* ] A start job is running for Configur…only root support (16s / no limit) [** ] A start job is running for Configur…only root support (16s / no limit)[ 51.202992] NFS: Registering the id_resolver key type [ 51.212803] Key type id_resolver registered [ 51.219805] Key type id_legacy registered [*** ] A start job is running for Configur…only root support (17s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started Login Service. [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg232-server login: [ 115.017221] libcfs: loading out-of-tree module taints kernel. [ 115.045974] Key type ._llcrypt registered [ 115.049552] Key type .llcrypt registered [ 115.163309] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [ 134.631692] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 135.945812] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 135.955647] alg: No test for adler32 (adler32-zlib) [ 137.265206] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [ 137.922312] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 139.679211] Key type lgssc registered [ 141.071936] Lustre: Echo OBD driver; http://www.lustre.org/ [ 156.967652] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 197.140482] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 207.878925] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 207.921756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 209.181567] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 209.203289] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 209.294383] Lustre: lustre-MDT0000: new disk, initializing [ 209.417870] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 209.431847] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 213.790664] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 227.539198] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 227.663782] Lustre: 6509:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 227.722464] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 227.726548] Lustre: Skipped 1 previous similar message [ 227.879685] Lustre: lustre-MDT0001: new disk, initializing [ 227.941504] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 227.989681] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 228.000353] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 232.493589] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 237.954519] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 248.799730] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 249.136816] Lustre: lustre-OST0000: new disk, initializing [ 249.141851] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 249.147741] Lustre: 8447:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 249.225465] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 250.477217] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 250.495251] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 250.593760] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 257.366378] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 273.522633] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 273.748818] Lustre: lustre-OST0001: new disk, initializing [ 273.760603] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 273.770447] Lustre: 9521:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 273.892384] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 281.599727] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 283.672542] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 283.684628] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 283.735901] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 295.507533] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 301.992460] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 308.924667] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing check_logdir /tmp/testlogs/ [ 314.554353] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing yml_node [ 320.910221] Lustre: DEBUG MARKER: Client: 2.17.58.39 [ 323.942963] Lustre: DEBUG MARKER: MDS: 2.17.58.39 [ 326.513414] Lustre: DEBUG MARKER: OSS: 2.17.58.39 [ 328.091673] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 17 04:55:03 EDT 2026 [ 346.805801] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 348.271783] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 365.543793] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 365.562687] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 365.581250] Lustre: Skipped 2 previous similar messages [ 370.592937] Lustre: server umount lustre-MDT0000 complete [ 374.709192] LustreError: 6499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789635350 with bad export cookie 9720495763347243537 [ 374.719072] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 374.730503] LustreError: 6499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 375.776060] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 375.780151] LustreError: 9429:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 375.808317] Lustre: Skipped 3 previous similar messages [ 375.849935] LustreError: 9429:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 375.886640] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 375.891053] Lustre: Skipped 1 previous similar message [ 379.810808] LustreError: 6515:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 379.813588] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 379.859285] Lustre: Skipped 1 previous similar message [ 384.930413] LustreError: 6514:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 384.930726] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 384.961980] LustreError: 6514:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 384.996882] Lustre: Skipped 1 previous similar message [ 389.090896] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 389.385610] Lustre: server umount lustre-MDT0001 complete [ 394.839837] Lustre: server umount lustre-OST0000 complete [ 399.769458] Lustre: server umount lustre-OST0001 complete [ 411.065791] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [ 429.614350] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 483.372346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 483.799445] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 483.889693] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 484.054330] Lustre: lustre-MDT0000: new disk, initializing [ 484.292933] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 484.330921] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 492.430152] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 507.801471] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 508.046467] Lustre: 15101:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 508.145195] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 508.153055] Lustre: Skipped 1 previous similar message [ 508.281436] Lustre: lustre-MDT0001: new disk, initializing [ 508.417876] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 508.486794] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 508.509687] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 514.472155] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 527.703264] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 536.077458] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 545.081921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 545.407594] Lustre: lustre-OST0000: new disk, initializing [ 545.420266] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 545.427939] Lustre: 17076:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 545.549271] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 547.060274] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 547.084321] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 547.314219] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 555.356121] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 566.493435] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 572.158735] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 572.511854] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 579.377665] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 579.714352] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 583.160727] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 583.172367] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 583.179422] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 583.198813] Lustre: Skipped 2 previous similar messages [ 583.212000] Lustre: Skipped 1 previous similar message [ 588.631025] Lustre: server umount lustre-OST0000 complete [ 595.429726] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 595.444226] LustreError: Skipped 1 previous similar message [ 595.452346] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 595.471611] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 595.480345] Lustre: Skipped 2 previous similar messages [ 600.482843] Lustre: server umount lustre-MDT0000 complete [ 603.625915] LustreError: 15108:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 603.671255] LustreError: 15108:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 605.598730] LustreError: 15900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789635581 with bad export cookie 9720495763347245070 [ 605.618817] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 605.627963] LustreError: 15900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 606.234621] Lustre: server umount lustre-MDT0001 complete [ 612.200944] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 04:59:46 (1789635586) [ 615.461407] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 617.879304] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 04:59:52 (1789635592) [ 626.791466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 627.220393] LustreError: 19487:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 627.278112] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 632.784245] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 637.414948] LustreError: 19488:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 637.429143] LustreError: 19488:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 641.796701] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 647.612337] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 657.614942] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 664.828585] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 672.807985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 679.421931] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 687.555702] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 707.044848] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 707.048640] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 707.061659] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 707.061668] Lustre: Skipped 2 previous similar messages [ 707.114189] Lustre: Skipped 2 previous similar messages [ 712.747416] Lustre: server umount lustre-OST0000 complete [ 717.280285] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 717.296980] Lustre: Skipped 1 previous similar message [ 723.552178] Lustre: server umount lustre-MDT0000 complete [ 727.211135] LustreError: 19467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789635703 with bad export cookie 9720495763347245952 [ 727.227229] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 727.229738] LustreError: 19467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 727.573836] Lustre: server umount lustre-MDT0001 complete [ 739.393260] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 742.302873] Key type lgssc unregistered [ 742.653623] LNet: 23056:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 742.662692] LNetError: 23056:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 742.685622] LNet: Removed LNI 192.168.202.132@tcp [ 743.584159] Key type .llcrypt unregistered [ 743.588355] Key type ._llcrypt unregistered [ 763.319818] Key type ._llcrypt registered [ 763.322491] Key type .llcrypt registered [ 763.473471] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 764.350963] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 764.465136] alg: No test for adler32 (adler32-zlib) [ 765.593849] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [ 765.861323] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 767.551209] Key type lgssc registered [ 768.584732] Lustre: Echo OBD driver; http://www.lustre.org/ [ 776.207382] loop: module loaded [ 776.258572] loop0: detected capacity change from 0 to 204800000 [ 776.405805] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 776.412616] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 776.586523] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 782.265869] loop0: detected capacity change from 0 to 204800000 [ 782.434819] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 782.445627] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 782.473666] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 787.267236] loop0: detected capacity change from 0 to 204800000 [ 793.012483] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 793.037487] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 794.323717] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 794.396610] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 794.521707] Lustre: lustre-MDT0000: new disk, initializing [ 794.617846] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 794.636844] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 799.062132] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 807.395290] loop1: detected capacity change from 0 to 204800000 [ 812.746391] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 812.963193] Lustre: lustre-OST07c6: new disk, initializing [ 812.966570] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 812.970599] Lustre: Skipped 1 previous similar message [ 812.974468] Lustre: 26215:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 813.028471] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 818.552256] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 822.865882] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 822.874292] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 822.911516] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 829.097164] Lustre: 26731:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 255 < left 264, rollback = 2 [ 829.104633] Lustre: 26731:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 829.108861] Lustre: 26731:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 829.113754] Lustre: 26731:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 829.119195] Lustre: 26731:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 829.123769] Lustre: 26731:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 846.414675] Lustre: 25206:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 846.424435] Lustre: 25206:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 846.440843] Lustre: 25206:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 846.457551] Lustre: 25206:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 846.475666] Lustre: 25206:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 846.485363] Lustre: 25206:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 851.154578] Lustre: 26731:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 851.161473] Lustre: 26731:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 851.166339] Lustre: 26731:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 851.173110] Lustre: 26731:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 851.178213] Lustre: 26731:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 851.182589] Lustre: 26731:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 851.187256] Lustre: 26731:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 851.192162] Lustre: 26731:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 851.198968] Lustre: 26731:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 851.208799] Lustre: 26731:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 851.214580] Lustre: 26731:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 851.219366] Lustre: 26731:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 854.360174] Lustre: 26731:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 854.367035] Lustre: 26731:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 854.372215] Lustre: 26731:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 854.374462] Lustre: 26731:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 854.381814] Lustre: 26731:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 854.392146] Lustre: 26731:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 854.396342] Lustre: 26731:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 854.402846] Lustre: 26731:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 854.409676] Lustre: 26731:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 854.417398] Lustre: 26731:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 854.424439] Lustre: 26731:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 854.432623] Lustre: 26731:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 858.864718] Lustre: 25204:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 858.872203] Lustre: 25204:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 858.882836] Lustre: 25204:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 858.890026] Lustre: 25204:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 858.898975] Lustre: 25204:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 858.908568] Lustre: 25204:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 858.914875] Lustre: 25204:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 858.929861] Lustre: 25204:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 858.933699] Lustre: 25204:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 858.939987] Lustre: 25204:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 858.945986] Lustre: 25204:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 858.952300] Lustre: 25204:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 869.796633] Lustre: 26731:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 869.805764] Lustre: 26731:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 869.811909] Lustre: 26731:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 869.816882] Lustre: 26731:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 869.819581] Lustre: 26731:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 869.825962] Lustre: 26731:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 869.831419] Lustre: 26731:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 869.839183] Lustre: 26731:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 869.845688] Lustre: 26731:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 869.850158] Lustre: 26731:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 869.854741] Lustre: 26731:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 869.861208] Lustre: 26731:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 920.040120] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 920.052472] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 920.068098] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 925.153715] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 930.271200] Lustre: lustre-OST07c6 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 930.276423] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 930.475482] Lustre: server umount lustre-OST07c6 complete [ 939.566707] Lustre: server umount lustre-MDT0000 complete [ 956.814473] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 959.485026] Key type lgssc unregistered [ 959.820195] LNet: 28775:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 959.830523] LNetError: 28775:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 960.870760] LNet: Removed LNI 192.168.202.132@tcp [ 961.660161] Key type .llcrypt unregistered [ 961.662191] Key type ._llcrypt unregistered [ 977.634595] Key type ._llcrypt registered [ 977.636353] Key type .llcrypt registered [ 977.781383] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 978.819529] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 978.908308] alg: No test for adler32 (adler32-zlib) [ 979.944734] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [ 980.199456] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 981.887278] Key type lgssc registered [ 983.071406] Lustre: Echo OBD driver; http://www.lustre.org/ [ 991.134698] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 991.151252] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 992.570431] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 996.756354] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1001.955721] LustreError: 30022:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1001.966217] LustreError: 30022:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1004.438249] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1004.671497] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1008.442933] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1014.856524] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1019.634147] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1025.971520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1026.248322] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1031.779483] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1037.563722] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 1039.757768] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1055.201499] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1055.209432] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1055.244492] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1058.277489] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1058.285147] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1058.297336] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1059.531101] Lustre: server umount lustre-OST0000 complete [ 1065.960908] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1065.966526] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1065.983499] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1065.990736] Lustre: Skipped 1 previous similar message [ 1068.513575] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1068.515812] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1068.545478] Lustre: Skipped 1 previous similar message [ 1073.632339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1078.239194] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1078.460580] Lustre: server umount lustre-MDT0000 complete [ 1078.753163] LustreError: 30021:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1078.767887] LustreError: 30021:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1081.594270] LustreError: 30003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789636057 with bad export cookie 1827381459412699297 [ 1081.601576] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1081.604775] LustreError: 30003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1081.875894] Lustre: server umount lustre-MDT0001 complete [ 1091.727979] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 1094.039630] Key type lgssc unregistered [ 1094.303649] LNet: 33756:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1094.318752] LNetError: 33756:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1095.333576] LNet: Removed LNI 192.168.202.132@tcp [ 1096.041896] Key type .llcrypt unregistered [ 1096.044049] Key type ._llcrypt unregistered [ 1114.370900] Key type ._llcrypt registered [ 1114.373827] Key type .llcrypt registered [ 1114.467701] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 1115.501974] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1115.606805] alg: No test for adler32 (adler32-zlib) [ 1116.669284] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [ 1116.986653] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 1118.673201] Key type lgssc registered [ 1119.754779] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1125.802208] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1125.826226] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1127.168327] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1131.379715] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1137.643663] LustreError: 35028:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1137.662037] LustreError: 35028:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1139.925677] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1140.240985] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1143.142283] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1149.370070] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1153.927838] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1159.024650] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1159.363271] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1164.599226] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1170.241424] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1172.724959] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1183.207078] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1183.213403] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1183.219308] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1183.253025] Lustre: Skipped 1 previous similar message [ 1188.323328] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1188.338789] Lustre: Skipped 2 previous similar messages [ 1189.190954] Lustre: server umount lustre-OST0000 complete [ 1193.441265] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1193.453919] Lustre: Skipped 1 previous similar message [ 1193.456962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1193.467516] Lustre: Skipped 1 previous similar message [ 1198.520479] Lustre: server umount lustre-MDT0000 complete [ 1198.561437] LustreError: 35023:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1198.577610] LustreError: 35023:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1201.788966] LustreError: 35008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789636178 with bad export cookie 741704098405411795 [ 1201.791509] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1201.803411] LustreError: 35008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1202.048210] Lustre: server umount lustre-MDT0001 complete [ 1210.398075] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 1212.460890] Key type lgssc unregistered [ 1212.671648] LNet: 38581:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1212.675371] LNetError: 38581:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1212.685591] LNet: Removed LNI 192.168.202.132@tcp [ 1213.202773] Key type .llcrypt unregistered [ 1213.204192] Key type ._llcrypt unregistered [ 1231.970395] Key type ._llcrypt registered [ 1231.973400] Key type .llcrypt registered [ 1232.059489] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 1232.706218] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1232.819922] alg: No test for adler32 (adler32-zlib) [ 1233.884664] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [ 1234.047567] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 1235.719242] Key type lgssc registered [ 1236.494976] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1242.262476] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1242.280744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1243.640276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1247.026528] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1252.833931] LustreError: 39973:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1252.848854] LustreError: 39973:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1253.933511] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1254.258978] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1257.933569] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1263.271284] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1266.857617] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1271.827506] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1272.219175] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1277.449361] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1282.895787] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1288.624148] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1298.873834] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.32@tcp (stopping) [ 1298.913713] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1298.924140] Lustre: Skipped 2 previous similar messages [ 1303.797528] Lustre: server umount lustre-MDT0000 complete [ 1303.964214] LustreError: 39969:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1306.807166] LustreError: 39954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789636283 with bad export cookie 11890729126784135363 [ 1306.818804] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1309.080300] LustreError: 42464:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1309.094466] LustreError: 42464:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 1309.106842] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.32@tcp (stopping) [ 1309.112846] Lustre: Skipped 3 previous similar messages [ 1309.153434] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1311.201992] LustreError: 42463:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1311.202599] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1311.213301] LustreError: 42463:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1311.221119] Lustre: Skipped 1 previous similar message [ 1313.221308] Lustre: server umount lustre-MDT0001 complete [ 1341.408294] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1341.552784] Lustre: server umount lustre-OST0000 complete [ 1353.979573] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 1355.875869] Key type lgssc unregistered [ 1356.121935] LNet: 43853:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1356.130225] LNetError: 43853:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1356.139284] LNet: Removed LNI 192.168.202.132@tcp [ 1356.745850] Key type .llcrypt unregistered [ 1356.748138] Key type ._llcrypt unregistered [ 1373.889250] Key type ._llcrypt registered [ 1373.892918] Key type .llcrypt registered [ 1373.986676] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 1374.879337] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1374.897645] alg: No test for adler32 (adler32-zlib) [ 1375.871404] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [ 1376.038625] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 1377.663553] Key type lgssc registered [ 1378.446585] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1384.116953] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1384.149947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1385.480070] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1388.789710] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1395.376802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1395.727923] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1399.381950] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1405.294619] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1409.383684] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1414.403736] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1414.663174] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1418.848967] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1425.216661] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1432.123666] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1440.225184] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1440.238171] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1440.257256] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1441.208214] Lustre: lustre-OST0000: Not available for connect from 192.168.202.32@tcp (stopping) [ 1442.277158] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1442.279041] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1442.297962] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1442.301658] Lustre: Skipped 1 previous similar message [ 1446.161172] Lustre: server umount lustre-OST0000 complete [ 1466.726808] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1466.729137] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1466.743765] Lustre: Skipped 1 previous similar message [ 1466.750727] Lustre: Skipped 1 previous similar message [ 1468.920903] Lustre: server umount lustre-MDT0000 complete [ 1471.723479] LustreError: 45226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789636448 with bad export cookie 14124503184609901852 [ 1471.741241] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1486.303211] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1486.604326] Lustre: server umount lustre-MDT0001 complete [ 1495.175837] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 1497.279572] Key type lgssc unregistered [ 1497.495476] LNet: 48968:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1497.500305] LNetError: 48968:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1497.512607] LNet: Removed LNI 192.168.202.132@tcp [ 1498.180982] Key type .llcrypt unregistered [ 1498.183055] Key type ._llcrypt unregistered [ 1514.707282] Key type ._llcrypt registered [ 1514.709996] Key type .llcrypt registered [ 1514.798788] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 1515.592560] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1515.605290] alg: No test for adler32 (adler32-zlib) [ 1516.634243] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [ 1516.807770] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 1518.439203] Key type lgssc registered [ 1519.353959] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1524.885271] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1524.895313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1526.187535] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1529.644467] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1536.490745] LustreError: 50359:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1536.502598] LustreError: 50359:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1537.343143] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1537.605917] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1541.320914] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1546.588326] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1551.011242] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1557.092597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1557.504242] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1562.962233] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1569.433378] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1570.805116] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1578.382617] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1581.244314] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1592.767427] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-17 5h16m07s [ 1593.960452] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1595.841904] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.32@tcp (stopping) [ 1596.388180] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1596.391424] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1596.403571] Lustre: Skipped 2 previous similar messages [ 1596.426780] Lustre: Skipped 2 previous similar messages [ 1597.474431] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1597.477537] Lustre: Skipped 2 previous similar messages [ 1601.760828] Lustre: server umount lustre-MDT0000 complete [ 1601.945119] LustreError: 50355:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1603.554385] LustreError: 51812:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1603.576589] LustreError: 51812:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1604.767478] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1608.097725] LustreError: 51812:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1610.253833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1610.356456] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1610.563034] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1610.629608] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1611.562991] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1614.534381] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1617.229135] Lustre: DEBUG MARKER: Wait for df (36651) ... [ 1618.482890] Lustre: DEBUG MARKER: done [ 1621.984230] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1621.993447] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1622.000196] Lustre: Skipped 1 previous similar message [ 1627.104806] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1627.112454] Lustre: Skipped 1 previous similar message [ 1627.815623] Lustre: server umount lustre-OST0000 complete [ 1632.229891] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1632.232606] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1632.245111] Lustre: Skipped 2 previous similar messages [ 1638.370179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1638.376334] Lustre: Skipped 5 previous similar messages [ 1645.023223] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1645.146181] Lustre: server umount lustre-MDT0000 complete [ 1648.116989] LustreError: 50340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789636624 with bad export cookie 11004357610569759420 [ 1648.123026] LustreError: 50340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1648.127123] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1648.459000] Lustre: server umount lustre-MDT0001 complete [ 1657.464868] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 1659.771539] Key type lgssc unregistered [ 1660.046584] LNet: 55283:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1660.054766] LNetError: 55283:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1660.067271] LNet: Removed LNI 192.168.202.132@tcp [ 1660.805594] Key type .llcrypt unregistered [ 1660.809110] Key type ._llcrypt unregistered [ 1692.190788] Key type ._llcrypt registered [ 1692.194742] Key type .llcrypt registered [ 1692.290929] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 1693.238955] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1693.277588] alg: No test for adler32 (adler32-zlib) [ 1694.350974] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [ 1694.550656] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 1696.215241] Key type lgssc registered [ 1697.013786] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1703.388965] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1703.409960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1704.682808] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1704.710319] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1704.948908] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1709.389681] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1717.297373] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1717.409895] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1717.435592] Lustre: 57470:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1717.709828] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1722.228947] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1728.103306] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1732.701526] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1739.092637] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1739.245579] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1739.422432] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1744.348614] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1749.630617] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1750.270879] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1755.691370] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-17 5h18m51s [ 1756.928684] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1762.347750] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1764.386656] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1764.455824] LustreError: 58925:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1784.775969] Lustre: lustre-MDT0000: Client 019fd488-2151-4620-9d41-a78acac4682a (at 192.168.202.32@tcp) reconnecting [ 1804.471146] LustreError: 58925:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1805.907947] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1807.590760] Lustre: DEBUG MARKER: done [ 1816.034498] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1816.039603] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1816.046814] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1816.549994] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1816.568529] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1816.571332] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1818.050737] Lustre: server umount lustre-OST0000 complete [ 1821.665254] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1821.667597] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1821.676310] Lustre: Skipped 1 previous similar message [ 1821.689441] Lustre: Skipped 1 previous similar message [ 1827.299930] Lustre: server umount lustre-MDT0000 complete [ 1830.608875] LustreError: 60228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789636806 with bad export cookie 8079419704894487963 [ 1830.629553] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1830.914177] Lustre: server umount lustre-MDT0001 complete [ 1839.248896] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 1841.458287] Key type lgssc unregistered [ 1841.681877] LNet: 61795:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1841.686711] LNetError: 61795:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1842.730886] LNet: Removed LNI 192.168.202.132@tcp [ 1843.442456] Key type .llcrypt unregistered [ 1843.448147] Key type ._llcrypt unregistered [ 1875.718637] Key type ._llcrypt registered [ 1875.721406] Key type .llcrypt registered [ 1875.842571] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 1876.760519] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1876.865964] alg: No test for adler32 (adler32-zlib) [ 1877.816644] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [ 1878.032579] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 1879.729088] Key type lgssc registered [ 1880.638436] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1883.407554] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1883.423501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1884.659374] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1884.673874] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1884.791580] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1885.536633] Lustre: Failing over lustre-MDT0000 [ 1885.836115] Lustre: server umount lustre-MDT0000 complete [ 1891.018895] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 05:21:06 (1789636866) [ 1896.018182] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1896.307473] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1899.279654] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1905.798542] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1905.856667] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1905.882953] Lustre: 64568:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1906.067161] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1909.771864] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1915.680352] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1921.050871] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1926.738975] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1926.874458] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1927.025274] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1931.810654] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1936.477368] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1937.957343] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1943.382030] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1949.940204] Lustre: server umount lustre-MDT0000 complete [ 1951.726547] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1951.727468] LustreError: 64575:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1951.738210] Lustre: Skipped 2 previous similar messages [ 1951.764842] LustreError: 64575:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 1952.760636] LustreError: 64559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789636929 with bad export cookie 5454226138213556658 [ 1952.761774] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1952.768073] LustreError: 64559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1953.067922] Lustre: server umount lustre-MDT0001 complete [ 1955.855917] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1959.471538] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1966.971177] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1973.215243] Lustre: 63491:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789636933/real 1789636933] req@00000000ea63f249 x1876570253857024/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789636949 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1973.242168] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1974.239439] Lustre: 63489:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789636934/real 1789636934] req@0000000094175e1d x1876570253857152/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789636950 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1978.399159] Lustre: 63489:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789636938/real 1789636938] req@00000000185c0328 x1876570253857280/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789636954 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1978.703280] LustreError: 66418:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1978.808709] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1978.916018] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1982.382684] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1982.882603] LustreError: 66417:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1982.897406] LustreError: 66417:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1983.903460] Lustre: 63490:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789636944/real 1789636944] req@00000000f20eb433 x1876570253857664/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789636960 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1983.931967] Lustre: 63490:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1987.935584] Lustre: 63489:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789636948/real 1789636948] req@00000000e29f91f7 x1876570253857920/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789636964 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1988.071136] LustreError: 66418:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1988.083996] LustreError: 66418:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1989.376838] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1989.710744] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1991.810842] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1993.461541] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1999.620723] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2004.263775] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2010.080385] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2010.086138] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2010.100764] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2012.644379] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2012.660537] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2015.183788] Lustre: server umount lustre-MDT0000 complete [ 2017.766034] LustreError: 66418:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2017.792554] LustreError: 66418:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2018.067700] LustreError: 64560:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789636994 with bad export cookie 5454226138213558933 [ 2018.075270] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2018.077229] LustreError: 64560:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2018.323493] Lustre: server umount lustre-MDT0001 complete [ 2023.184626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2030.302579] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2039.071214] Lustre: 63490:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789636999/real 1789636999] req@000000003ea3f5a9 x1876570253883008/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789637015 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2039.106564] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2039.115492] Lustre: Skipped 1 previous similar message [ 2043.541450] LustreError: 69069:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2043.602613] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2043.650883] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2048.008139] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2054.782795] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2058.671344] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2060.260380] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2060.273479] Lustre: Skipped 1 previous similar message [ 2064.041823] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2068.463930] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2068.505478] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2073.568090] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2073.574383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2073.578834] Lustre: Skipped 1 previous similar message [ 2073.590018] Lustre: Skipped 4 previous similar messages [ 2078.689470] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2078.700605] Lustre: Skipped 2 previous similar messages [ 2079.054638] Lustre: server umount lustre-MDT0000 complete [ 2082.034782] LustreError: 64560:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789637058 with bad export cookie 5454226138213560312 [ 2082.035957] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2082.040763] LustreError: 64560:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2082.279965] Lustre: server umount lustre-MDT0001 complete [ 2087.026972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2089.512328] Lustre: DEBUG MARKER: files compared the same [ 2101.207638] Lustre: server umount lustre-OST0000 complete [ 2113.644764] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 2115.762787] Key type lgssc unregistered [ 2116.004993] LNet: 74288:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2116.010591] LNetError: 74288:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2116.031748] LNet: Removed LNI 192.168.202.132@tcp [ 2116.650625] Key type .llcrypt unregistered [ 2116.652748] Key type ._llcrypt unregistered [ 2134.020495] Key type ._llcrypt registered [ 2134.021965] Key type .llcrypt registered [ 2134.094575] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 2134.885709] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2134.948284] alg: No test for adler32 (adler32-zlib) [ 2135.932229] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [ 2136.138526] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 2137.785567] Key type lgssc registered [ 2138.616439] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2144.165988] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2144.193980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2145.530064] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2148.327873] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug +malloc all [ 2154.232270] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2154.401748] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2157.248239] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug +malloc all [ 2162.380864] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2166.298753] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2170.566885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2170.720092] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 2170.783877] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2174.930692] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug +malloc all [ 2175.982458] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2180.235874] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2189.281237] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2189.283281] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2189.287439] Lustre: Skipped 1 previous similar message [ 2189.300064] Lustre: Skipped 1 previous similar message [ 2193.403317] Lustre: server umount lustre-OST0000 complete [ 2199.529677] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2199.536691] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2199.541075] Lustre: Skipped 1 previous similar message [ 2199.551961] Lustre: Skipped 1 previous similar message [ 2204.641776] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2204.650924] Lustre: Skipped 1 previous similar message [ 2208.737237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2208.740995] Lustre: Skipped 1 previous similar message [ 2210.271210] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2210.452209] Lustre: server umount lustre-MDT0000 complete [ 2213.531754] LustreError: 75661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789637189 with bad export cookie 7338099596497228830 [ 2213.537533] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2213.541183] LustreError: 75661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2213.859942] Lustre: server umount lustre-MDT0001 complete [ 2221.475358] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 2223.373652] Key type lgssc unregistered [ 2223.617726] LNet: 79210:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2223.621883] LNetError: 79210:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2223.634391] LNet: Removed LNI 192.168.202.132@tcp [ 2224.218400] Key type .llcrypt unregistered [ 2224.220222] Key type ._llcrypt unregistered [ 2241.011364] Key type ._llcrypt registered [ 2241.013767] Key type .llcrypt registered [ 2241.108733] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 2242.011697] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2242.034424] alg: No test for adler32 (adler32-zlib) [ 2243.019450] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [ 2243.243215] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 2244.935561] Key type lgssc registered [ 2246.005773] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2251.960675] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2251.980703] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2267.615419] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2272.736406] LustreError: 80575:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.132@tcp: failed processing log, type 4: rc = -110 [ 2299.359359] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2303.513743] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2309.254577] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2314.975759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2315.043762] Lustre: *** cfs_fail_loc=706, val=0*** [ 2315.261579] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2315.297246] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2318.202117] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2320.355246] LustreError: 80601:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2320.371780] LustreError: 80601:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2324.041709] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2324.330291] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2327.639747] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2333.041936] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2337.252595] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2339.811448] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2339.816720] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2339.820646] Lustre: Skipped 1 previous similar message [ 2339.829150] Lustre: Skipped 1 previous similar message [ 2344.931552] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2344.937944] Lustre: Skipped 1 previous similar message [ 2345.151842] Lustre: server umount lustre-OST0000 complete [ 2350.048784] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2350.053657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2350.058919] Lustre: Skipped 1 previous similar message [ 2355.169116] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2355.174019] Lustre: Skipped 2 previous similar messages [ 2360.288984] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2360.299259] Lustre: Skipped 1 previous similar message [ 2362.335277] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2362.718610] Lustre: server umount lustre-MDT0000 complete [ 2365.416380] LustreError: 81664:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2365.437312] LustreError: 81664:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2366.279922] LustreError: 80583:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789637342 with bad export cookie 2345675573624991871 [ 2366.291014] LustreError: 80583:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2366.297184] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2366.700991] Lustre: server umount lustre-MDT0001 complete [ 2376.357887] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 2378.466514] Key type lgssc unregistered [ 2378.687612] LNet: 84190:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2378.702293] LNetError: 84190:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2378.716802] LNet: Removed LNI 192.168.202.132@tcp [ 2379.318966] Key type .llcrypt unregistered [ 2379.321030] Key type ._llcrypt unregistered [ 2396.822867] Key type ._llcrypt registered [ 2396.824646] Key type .llcrypt registered [ 2396.889311] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 2397.673809] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2397.691037] alg: No test for adler32 (adler32-zlib) [ 2398.697441] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [ 2398.867690] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 2400.503268] Key type lgssc registered [ 2401.395970] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2408.742633] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2408.774955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2415.354861] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2424.680042] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2425.442041] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2430.436704] LustreError: 86270:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2430.448764] LustreError: 86270:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2430.630899] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2435.575825] LustreError: 86273:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2435.588749] LustreError: 86273:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2437.596969] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2437.824614] LustreError: 86270:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2437.837558] LustreError: 86270:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2437.854709] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2443.152932] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2443.237496] LustreError: 86274:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2443.245206] LustreError: 86274:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2448.355400] LustreError: 86273:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2448.378672] LustreError: 86273:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2449.646520] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2449.658631] Lustre: MGS: Client d0e89363-56ac-43a0-bdc4-6934fca95ec7 (at 0@lo) reconnecting [ 2449.674888] LustreError: 86284:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2449.896325] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2449.966424] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2453.188929] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2465.248391] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2465.261511] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2465.280230] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2466.282610] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2466.285766] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2466.286409] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2470.026427] Lustre: server umount lustre-OST0000 complete [ 2476.512781] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2476.517636] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2476.526312] Lustre: Skipped 1 previous similar message [ 2476.534195] Lustre: Skipped 2 previous similar messages [ 2479.725864] Lustre: server umount lustre-MDT0000 complete [ 2481.634894] LustreError: 86274:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2481.650501] LustreError: 86274:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2482.869241] Lustre: server umount lustre-MDT0001 complete [ 2485.489580] LustreError: 88883:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x0:0x0].0x0 (00000000dd96d3c5) refcount nonzero (0) after lock cleanup; forcing cleanup. [ 2485.530448] Lustre: server umount MGS complete [ 2493.173608] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 05:31:08 (1789637468) [ 2512.227442] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [ 2519.324737] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 2543.404277] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2546.838905] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2554.590388] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2554.761026] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2554.812856] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2554.871923] Lustre: lustre-MDT0001: new disk, initializing [ 2554.922497] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2554.942076] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2554.946914] Lustre: 93434:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2554.956477] LustreError: 93434:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2558.378795] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2567.105665] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2567.333839] Lustre: lustre-OST0000: new disk, initializing [ 2567.336693] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2567.345351] Lustre: 94199:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2567.415562] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2572.792952] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2577.443111] Lustre: 94716:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000f1cd1ef8 x1876570800166528/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1789637569 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2579.660562] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2585.394585] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2585.404353] Lustre: MGS: Client 92903098-37bc-4e7c-bdd5-ff4dfd491155 (at 0@lo) reconnecting [ 2585.421253] LustreError: 93420:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2585.474971] Lustre: 92705:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2585.512537] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2585.586437] Lustre: lustre-MDT0000: new disk, initializing [ 2585.744153] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2585.777859] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2586.531139] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2586.848033] Lustre: 95268:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000003f27b16e x1876570800172672/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1789637579 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2589.952438] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2591.909954] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2591.919516] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2591.994443] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2612.712162] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2612.718056] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2612.733817] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2617.834684] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2617.838615] Lustre: Skipped 2 previous similar messages [ 2618.447656] Lustre: server umount lustre-OST0000 complete [ 2622.945766] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2622.948272] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2622.969917] Lustre: Skipped 2 previous similar messages [ 2628.065454] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2628.077631] Lustre: Skipped 3 previous similar messages [ 2635.231166] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2635.331528] Lustre: server umount lustre-MDT0000 complete [ 2643.791776] Lustre: server umount lustre-MDT0001 complete [ 2645.704618] Lustre: server umount MGS complete [ 2651.430962] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 05:33:47 (1789637627) [ 2660.747572] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 2662.497728] Key type lgssc unregistered [ 2662.690552] LNet: 97762:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2662.696944] LNetError: 97762:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2663.719964] LNet: Removed LNI 192.168.202.132@tcp [ 2664.355301] Key type .llcrypt unregistered [ 2664.357518] Key type ._llcrypt unregistered [ 2674.515655] Key type ._llcrypt registered [ 2674.520781] Key type .llcrypt registered [ 2674.601171] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 2675.180440] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2675.238202] alg: No test for adler32 (adler32-zlib) [ 2676.181816] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [ 2676.336185] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 2677.951303] Key type lgssc registered [ 2678.591264] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2683.730201] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2683.740509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2685.030626] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2688.946491] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2695.658131] LustreError: 98958:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2695.675018] LustreError: 98958:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2696.584886] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2696.960607] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2701.994761] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2713.239171] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2718.679483] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2726.474765] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2726.990276] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2730.049194] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 2736.707680] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2745.238799] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2755.555630] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2755.588883] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2755.615185] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2757.609462] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2757.611181] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2757.641866] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2757.692147] Lustre: Skipped 1 previous similar message [ 2760.870672] Lustre: server umount lustre-OST0000 complete [ 2767.842579] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2767.863653] Lustre: Skipped 1 previous similar message [ 2767.874986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2767.896411] Lustre: Skipped 1 previous similar message [ 2772.962696] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2772.975087] Lustre: Skipped 1 previous similar message [ 2773.820649] Lustre: server umount lustre-MDT0000 complete [ 2778.081065] LustreError: 99952:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2778.106363] LustreError: 99952:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2780.236101] LustreError: 98939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789637756 with bad export cookie 11291620334892333465 [ 2780.245144] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2780.258039] LustreError: 98939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2780.642247] Lustre: server umount lustre-MDT0001 complete [ 2796.258317] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 2799.542049] Key type lgssc unregistered [ 2799.905640] LNet: 102493:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2799.923066] LNetError: 102493:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2799.940676] LNet: Removed LNI 192.168.202.132@tcp [ 2800.836522] Key type .llcrypt unregistered [ 2800.846418] Key type ._llcrypt unregistered [ 2816.854887] Key type ._llcrypt registered [ 2816.857392] Key type .llcrypt registered [ 2816.955696] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 2818.446153] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2818.519197] alg: No test for adler32 (adler32-zlib) [ 2819.596650] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [ 2819.843646] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 2821.600240] Key type lgssc registered [ 2822.832623] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2829.021886] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2829.029480] LustreError: 103574:0:(tgt_mount.c:2569:server_fill_super()) cfs_race id 716 sleeping [ 2829.043849] LustreError: 103575:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 waking [ 2829.047719] LustreError: 103574:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 awake: rc=4992 [ 2829.056762] LustreError: 103574:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2829.061886] LustreError: 103574:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2829.063423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2830.559121] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2837.908397] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2838.210275] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2842.843509] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2848.239930] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing lsmod [ 2850.655919] LustreError: 104781:0:(tgt_mount.c:2569:server_fill_super()) cfs_race id 716 sleeping [ 2850.675133] LustreError: 104782:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 waking [ 2850.678814] LustreError: 104781:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 awake: rc=4986 [ 2850.687780] LustreError: 104781:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2850.697309] LustreError: 104781:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2850.705419] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2851.125445] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2853.857410] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2853.876083] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2853.885309] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2856.422659] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2856.422959] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2856.442152] Lustre: Skipped 1 previous similar message [ 2856.450960] Lustre: Skipped 2 previous similar messages [ 2861.537725] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2863.597596] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2863.602517] Lustre: Skipped 3 previous similar messages [ 2868.191776] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2868.706373] LustreError: 103609:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2868.718062] LustreError: 103609:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2868.766854] Lustre: server umount lustre-MDT0000 complete [ 2872.668282] LustreError: 103589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789637848 with bad export cookie 7737184118427830702 [ 2872.669553] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2872.684786] LustreError: 103589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2873.837570] LustreError: 104817:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2873.837640] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2873.874166] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2873.887449] LustreError: 104044:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000e783cdbb x1876571241600000/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2874.025306] Lustre: server umount lustre-MDT0001 complete [ 2880.194665] Lustre: server umount lustre-OST0000 complete [ 2890.268956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2890.773291] LustreError: 105848:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2890.849651] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2895.508194] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2895.845593] LustreError: 105849:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2900.977099] LustreError: 105848:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2905.843908] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2906.087512] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2906.244581] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2911.181885] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2919.420185] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2924.820446] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2931.297456] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2931.597637] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2936.760206] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 2937.819801] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2945.918897] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2957.280928] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2957.297211] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2957.312161] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2961.381920] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2961.382782] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2963.306266] Lustre: server umount lustre-OST0000 complete [ 2967.521245] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2971.617777] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2971.636221] Lustre: Skipped 1 previous similar message [ 2973.836201] Lustre: server umount lustre-MDT0000 complete [ 2976.743075] LustreError: 105849:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2976.753785] LustreError: 105849:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2977.475195] LustreError: 105828:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789637953 with bad export cookie 7737184118427831388 [ 2977.501682] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2978.115675] Lustre: server umount lustre-MDT0001 complete [ 2990.639847] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 2993.305346] Key type lgssc unregistered [ 2993.544329] LNet: 109387:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2993.556296] LNetError: 109387:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2994.600103] LNet: Removed LNI 192.168.202.132@tcp [ 2995.874098] Key type .llcrypt unregistered [ 2995.875874] Key type ._llcrypt unregistered [ 3024.404597] Key type ._llcrypt registered [ 3024.420555] Key type .llcrypt registered [ 3024.671964] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 3026.322670] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3026.363196] alg: No test for adler32 (adler32-zlib) [ 3027.825392] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [ 3028.263683] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 3030.049772] Key type lgssc registered [ 3031.660734] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3041.126931] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3041.149307] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3042.700177] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3047.372829] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3052.002134] LustreError: 110815:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3052.024607] LustreError: 110815:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3057.124588] LustreError: 110816:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3057.244841] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3057.606309] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3062.081980] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3070.232696] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3075.814304] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3083.089331] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3083.482311] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3087.622289] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3089.985756] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3098.075903] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3107.680418] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3111.639113] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3117.536790] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3117.543967] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3117.565446] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3117.599898] Lustre: Skipped 1 previous similar message [ 3122.664300] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3122.669236] Lustre: Skipped 1 previous similar message [ 3127.784258] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3127.794783] Lustre: Skipped 1 previous similar message [ 3131.359195] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3131.510407] Lustre: server umount lustre-OST0000 complete [ 3138.022392] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3138.034984] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3138.037718] Lustre: Skipped 1 previous similar message [ 3143.136387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3143.139381] Lustre: Skipped 1 previous similar message [ 3150.303276] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3150.664974] Lustre: server umount lustre-MDT0000 complete [ 3153.377072] LustreError: 110812:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3153.401415] LustreError: 110812:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3155.901714] LustreError: 110796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789638132 with bad export cookie 12383116427756195805 [ 3155.923153] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3156.616645] Lustre: server umount lustre-MDT0001 complete [ 3168.632733] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 3171.363960] Key type lgssc unregistered [ 3171.637046] LNet: 114455:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3171.652697] LNetError: 114455:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3171.679843] LNet: Removed LNI 192.168.202.132@tcp [ 3172.469377] Key type .llcrypt unregistered [ 3172.471924] Key type ._llcrypt unregistered [ 3191.966802] Key type ._llcrypt registered [ 3191.969963] Key type .llcrypt registered [ 3192.169304] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 3194.637620] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3195.061323] alg: No test for adler32 (adler32-zlib) [ 3196.304437] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [ 3196.687511] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 3198.535694] Key type lgssc registered [ 3200.396900] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3211.010768] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3211.077405] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3212.802571] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3218.065585] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3222.001288] LustreError: 115673:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3222.035683] LustreError: 115673:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3227.110261] LustreError: 115674:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3228.528625] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3229.156693] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3234.665609] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3245.006222] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3253.612986] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3261.793213] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3262.366565] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3269.628725] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3278.167716] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3281.862295] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3295.200164] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3295.209736] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3295.241562] LustreError: 118048:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3295.391957] Lustre: server umount lustre-OST0000 complete [ 3300.832333] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3300.838162] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3300.859768] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3302.369241] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3302.370730] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3307.517486] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3307.521352] Lustre: Skipped 2 previous similar messages [ 3309.814854] Lustre: server umount lustre-MDT0000 complete [ 3312.610529] LustreError: 115674:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3312.653473] LustreError: 115674:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3314.217364] LustreError: 118197:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789638290 with bad export cookie 18207929423521107835 [ 3314.223370] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3314.228975] LustreError: 118197:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3314.590612] Lustre: server umount lustre-MDT0001 complete [ 3330.358819] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 3333.774476] Key type lgssc unregistered [ 3334.227824] LNet: 119250:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3334.238567] LNetError: 119250:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3334.287505] LNet: Removed LNI 192.168.202.132@tcp [ 3335.162151] Key type .llcrypt unregistered [ 3335.166820] Key type ._llcrypt unregistered [ 3362.768774] Key type ._llcrypt registered [ 3362.771065] Key type .llcrypt registered [ 3362.929560] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 3364.091824] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3364.214233] alg: No test for adler32 (adler32-zlib) [ 3365.320813] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [ 3365.624464] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 3367.359236] Key type lgssc registered [ 3368.528946] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3376.969077] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3377.019160] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3378.653052] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3384.075317] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3387.875143] LustreError: 120664:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3387.910818] LustreError: 120664:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3392.996767] LustreError: 120663:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3393.602938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3394.006710] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3399.898477] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3408.474895] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3414.598081] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3421.019502] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3421.326812] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3423.420400] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3427.410827] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3436.163700] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3445.998612] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3451.503188] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3451.515731] Lustre: Skipped 2 previous similar messages [ 3456.081529] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3456.109375] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3457.022720] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3457.025814] Lustre: Skipped 2 previous similar messages [ 3465.359114] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3465.367079] Lustre: Skipped 1 previous similar message [ 3469.926228] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.32@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3469.936943] Lustre: Skipped 2 previous similar messages [ 3472.737574] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.32@tcp 192.168.0.[2,10]@tcp [ 3480.547991] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.32/24@tcp in log lustre-MDT0000 [ 3480.554390] Lustre: Skipped 2 previous similar messages [ 3481.498806] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.32/24@tcp [ 3481.504143] Lustre: Skipped 1 previous similar message [ 3490.273447] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3490.276190] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3490.284348] LustreError: Skipped 1 previous similar message [ 3490.288286] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3490.319701] Lustre: Skipped 1 previous similar message [ 3495.411647] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3495.424420] Lustre: Skipped 2 previous similar messages [ 3496.505678] Lustre: server umount lustre-OST0000 complete [ 3501.539584] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3501.553722] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3501.575544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3505.637060] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3505.641163] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3510.753811] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3510.768986] Lustre: Skipped 2 previous similar messages [ 3515.359644] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3515.907556] LustreError: 120659:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3516.011526] Lustre: server umount lustre-MDT0000 complete [ 3520.035502] LustreError: 120658:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3520.092112] LustreError: 120658:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3522.127892] LustreError: 120644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789638498 with bad export cookie 11486555873823175388 [ 3522.146456] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3522.151951] LustreError: 120644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3522.917908] Lustre: server umount lustre-MDT0001 complete [ 3534.880377] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 3537.726826] Key type lgssc unregistered [ 3538.134451] LNet: 125561:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3538.154026] LNetError: 125561:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3539.174795] LNet: Removed LNI 192.168.202.132@tcp [ 3540.441791] Key type .llcrypt unregistered [ 3540.443908] Key type ._llcrypt unregistered [ 3577.841347] Key type ._llcrypt registered [ 3577.843163] Key type .llcrypt registered [ 3577.926633] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 3579.624071] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3579.672792] alg: No test for adler32 (adler32-zlib) [ 3580.684148] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [ 3581.049716] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 3582.871611] Key type lgssc registered [ 3584.232677] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3592.076493] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3592.100249] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3593.529389] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3593.554653] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.32/24@tcp [ 3593.564138] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3599.041865] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3602.924905] LustreError: 127122:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3602.960536] LustreError: 127122:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3607.177269] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3607.466514] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3607.506205] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.32/24@tcp [ 3607.553994] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3612.021300] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3619.589973] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3624.859356] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3631.211526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3631.672281] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3637.854560] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 3638.219342] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3647.042642] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3663.335245] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3663.339321] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3663.343216] LustreError: Skipped 1 previous similar message [ 3663.344948] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3663.360451] Lustre: Skipped 1 previous similar message [ 3663.383824] Lustre: Skipped 1 previous similar message [ 3665.205989] Lustre: server umount lustre-OST0000 complete [ 3673.570581] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3673.586820] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3673.603019] Lustre: Skipped 1 previous similar message [ 3673.636935] Lustre: Skipped 1 previous similar message [ 3676.353191] Lustre: server umount lustre-MDT0000 complete [ 3678.691487] LustreError: 127122:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3678.710461] LustreError: 127122:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3680.351707] LustreError: 127102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789638656 with bad export cookie 15924424844382373611 [ 3680.354394] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3680.361879] LustreError: 127102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3680.761849] Lustre: server umount lustre-MDT0001 complete [ 3693.511199] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 3696.354715] Key type lgssc unregistered [ 3696.626613] LNet: 130708:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3696.634401] LNetError: 130708:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3696.659560] LNet: Removed LNI 192.168.202.132@tcp [ 3697.534927] Key type .llcrypt unregistered [ 3697.536368] Key type ._llcrypt unregistered [ 3725.869414] Key type ._llcrypt registered [ 3725.871228] Key type .llcrypt registered [ 3725.969503] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [ 3740.305714] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 3741.656227] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3741.693166] alg: No test for adler32 (adler32-zlib) [ 3742.871322] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [ 3743.162183] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 3744.823254] Key type lgssc registered [ 3746.187830] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3783.539137] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3783.597242] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3784.840897] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3784.874053] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3784.946762] Lustre: lustre-MDT0000: new disk, initializing [ 3785.068655] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3785.110637] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3790.377915] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3801.355557] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3801.496950] Lustre: 134101:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3801.566541] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3801.571592] Lustre: Skipped 1 previous similar message [ 3801.673702] Lustre: lustre-MDT0001: new disk, initializing [ 3801.859873] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3801.939599] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3801.959935] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3806.998887] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3816.543426] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3822.480774] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3830.100275] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3830.441393] Lustre: lustre-OST0000: new disk, initializing [ 3830.449386] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3830.467258] Lustre: 136071:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3830.602742] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3836.513278] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3836.520383] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3836.630427] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3836.930594] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3846.793890] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3851.565508] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3851.732455] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3857.093786] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3857.287257] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3862.499566] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3862.501885] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3862.502903] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3862.529370] Lustre: Skipped 1 previous similar message [ 3865.794088] Lustre: server umount lustre-OST0000 complete [ 3872.738360] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3872.771199] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3872.774016] Lustre: Skipped 1 previous similar message [ 3876.056589] Lustre: server umount lustre-MDT0000 complete [ 3880.688448] LustreError: 134094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789638856 with bad export cookie 612942072577901942 [ 3880.716086] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3880.726131] LustreError: 134094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3881.423164] Lustre: server umount lustre-MDT0001 complete [ 3890.568495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3890.924914] LustreError: 138287:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3890.945468] LustreError: 138287:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 3890.985707] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3895.330549] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3896.293691] LustreError: 138288:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3901.418881] LustreError: 138287:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3903.289050] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3903.588925] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3907.704385] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3915.251278] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3920.628757] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3927.296212] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3927.802075] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3933.418713] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3940.755802] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3945.537487] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3945.844932] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3950.085915] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3950.294724] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3960.429033] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3960.639821] Lustre: lustre-OST0001: new disk, initializing [ 3960.644617] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3960.657387] Lustre: 141663:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3960.782849] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3968.178833] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3968.195937] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3968.372116] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3968.379240] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3977.876947] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3978.109944] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3982.897866] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3983.121919] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3988.230398] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff99dec2a82000.ost_server_uuid 50 [ 3990.025633] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff99dec2a82000.ost_server_uuid in IDLE state after 0 sec [ 3999.200146] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3999.208205] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3999.218832] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3999.223442] Lustre: Skipped 2 previous similar messages [ 3999.232190] Lustre: Skipped 1 previous similar message [ 4004.334646] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4004.340136] Lustre: Skipped 1 previous similar message [ 4005.397890] Lustre: server umount lustre-OST0001 complete [ 4009.443134] LustreError: 140111:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4009.444759] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4009.468788] LustreError: 140111:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4009.493366] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4009.516230] Lustre: Skipped 1 previous similar message [ 4014.598772] LustreError: 140587:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4014.648850] LustreError: 140587:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4015.908746] Lustre: server umount lustre-OST0000 complete [ 4021.224213] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4021.227679] LustreError: Skipped 1 previous similar message [ 4021.247572] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4021.284520] Lustre: Skipped 1 previous similar message [ 4021.300771] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4021.312668] Lustre: Skipped 2 previous similar messages [ 4027.156169] Lustre: server umount lustre-MDT0000 complete [ 4029.921274] LustreError: 138287:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4029.941501] LustreError: 138287:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4031.578898] LustreError: 138268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789639007 with bad export cookie 612942072577902824 [ 4031.580061] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4031.585865] LustreError: 138268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4031.964992] Lustre: server umount lustre-MDT0001 complete [ 4058.083864] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 4061.440338] Key type lgssc unregistered [ 4061.851859] LNet: 144665:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4061.868975] LNetError: 144665:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4062.960634] LNet: Removed LNI 192.168.202.132@tcp [ 4064.162590] Key type .llcrypt unregistered [ 4064.167869] Key type ._llcrypt unregistered [ 4102.522316] Key type ._llcrypt registered [ 4102.523487] Key type .llcrypt registered [ 4102.607382] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [ 4114.321688] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 4115.211147] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4115.276137] alg: No test for adler32 (adler32-zlib) [ 4116.269666] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [ 4116.435443] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 4118.087330] Key type lgssc registered [ 4118.973628] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4147.405975] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4147.424726] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4148.591325] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4148.609154] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4148.668226] Lustre: lustre-MDT0000: new disk, initializing [ 4148.717681] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4148.732382] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4152.331600] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 4160.927794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4161.007416] Lustre: 148792:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4161.068272] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4161.071550] Lustre: Skipped 1 previous similar message [ 4161.130182] Lustre: lustre-MDT0001: new disk, initializing [ 4161.195527] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4161.221055] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4161.236958] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4164.265194] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 4171.437672] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4175.748906] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4181.200825] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4181.517870] Lustre: lustre-OST0000: new disk, initializing [ 4181.521085] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4181.527881] Lustre: 150763:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4181.622716] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4186.607628] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 4190.237362] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4190.244992] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4190.317690] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4193.774885] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4198.325434] Lustre: Failing over lustre-OST0000 [ 4198.495261] Lustre: server umount lustre-OST0000 complete [ 4215.469101] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4215.699594] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4215.721299] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4215.734901] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4215.756664] Lustre: 147017:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789639176/real 1789639176] req@00000000e76647ae x1876572600564608/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1789639192 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4216.735563] Lustre: 147016:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789639176/real 1789639176] req@00000000fc827b70 x1876572600564480/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789639192 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4216.762712] Lustre: 147016:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 4216.772678] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4220.895168] Lustre: 147017:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789639181/real 1789639181] req@0000000090c91862 x1876572600565632/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789639197 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4220.921555] Lustre: 147017:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4220.957715] Lustre: lustre-OST0000: Recovery over after 0:04, of 3 clients 3 recovered and 0 were evicted. [ 4220.959913] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4221.031213] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 4224.138280] Lustre: Failing over lustre-MDT0000 [ 4224.399264] Lustre: server umount lustre-MDT0000 complete [ 4226.018244] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4226.030278] Lustre: Skipped 3 previous similar messages [ 4227.039326] Lustre: 147018:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789639186/real 1789639186] req@000000002f27dac4 x1876572600566784/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789639202 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4231.140718] LustreError: 148799:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4231.151510] LustreError: 148799:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 4232.091966] LustreError: 148801:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4236.256342] LustreError: 148804:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4236.269445] LustreError: 148804:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4240.953305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4241.039802] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4241.231261] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4241.283048] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4242.346630] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4244.402234] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 4246.506909] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4246.510945] Lustre: Skipped 1 previous similar message [ 4246.523142] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4246.584220] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4251.623985] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4251.626299] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4251.637719] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4255.605335] Lustre: server umount lustre-OST0000 complete [ 4261.857734] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4261.861029] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4261.862163] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4261.862169] Lustre: Skipped 1 previous similar message [ 4261.866794] Lustre: Skipped 2 previous similar messages [ 4264.269904] Lustre: server umount lustre-MDT0000 complete [ 4266.713771] LustreError: 148784:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789639242 with bad export cookie 575738390251309786 [ 4266.716164] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4266.721371] LustreError: 148784:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4266.971591] Lustre: server umount lustre-MDT0001 complete [ 4274.509216] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 4276.557453] Key type lgssc unregistered [ 4276.764573] LNet: 154414:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4276.767736] LNetError: 154414:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4276.779088] LNet: Removed LNI 192.168.202.132@tcp [ 4277.318669] Key type .llcrypt unregistered [ 4277.323868] Key type ._llcrypt unregistered [ 4305.427958] Key type ._llcrypt registered [ 4305.430966] Key type .llcrypt registered [ 4305.521000] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 4306.277170] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4306.310193] alg: No test for adler32 (adler32-zlib) [ 4307.256649] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [ 4307.426853] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 4309.063248] Key type lgssc registered [ 4309.798417] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4318.090042] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4318.130958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4319.662203] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4327.098261] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 4328.934318] LustreError: 155939:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4328.944053] LustreError: 155939:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4334.056639] LustreError: 155938:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4339.170105] LustreError: 155939:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4340.630995] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4344.288940] LustreError: 155938:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4346.771750] LustreError: 155967:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4346.812078] LustreError: 155967:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4346.843779] LustreError: 155967:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 28, retries 0, failed: rc = -5 [ 4347.564415] Lustre: server umount lustre-MDT0000 complete [ 4371.478736] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [ 4381.770238] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 4427.751100] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4428.011626] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4428.068054] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4428.200211] Lustre: lustre-MDT0000: new disk, initializing [ 4428.317925] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4428.334856] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4432.938349] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 4443.562797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4443.791931] Lustre: 159292:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4443.888321] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4443.893877] Lustre: Skipped 1 previous similar message [ 4444.012495] Lustre: lustre-MDT0001: new disk, initializing [ 4444.174608] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4444.209641] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4444.221994] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4448.370255] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 4457.303310] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4462.187455] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4469.332597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4469.796981] Lustre: lustre-OST0000: new disk, initializing [ 4469.802253] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4469.812126] Lustre: 161259:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4469.984663] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4476.137180] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 4479.109206] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4479.121400] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4479.218230] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4485.226768] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4488.988304] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4490.653443] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 06:04:25 (1789639465) [ 4720.096368] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4720.102255] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4720.132183] Lustre: Skipped 1 previous similar message [ 4720.153301] Lustre: Skipped 2 previous similar messages [ 4725.219070] LustreError: 159303:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4725.262596] LustreError: 159303:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4725.390834] Lustre: server umount lustre-MDT0000 complete [ 4730.596130] LustreError: 159283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789639706 with bad export cookie 11330797652304846493 [ 4730.606970] LustreError: 159283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4730.611208] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4735.457775] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4735.459928] LustreError: 160661:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4735.481707] Lustre: Skipped 1 previous similar message [ 4735.482435] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4735.520983] LustreError: 160661:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4737.054389] Lustre: server umount lustre-MDT0001 complete [ 4747.459149] Lustre: server umount lustre-OST0000 complete [ 4758.754439] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [ 4772.238835] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 4813.363550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4813.652894] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4813.680628] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4813.754932] Lustre: lustre-MDT0000: new disk, initializing [ 4813.811800] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4813.828177] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4819.192330] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 4832.087848] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4832.271664] Lustre: 165917:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4832.320131] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4832.323949] Lustre: Skipped 1 previous similar message [ 4832.389353] Lustre: lustre-MDT0001: new disk, initializing [ 4832.476845] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4832.504074] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4832.510187] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4837.964682] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 4848.477944] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4854.397063] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4862.020967] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4862.279427] Lustre: lustre-OST0000: new disk, initializing [ 4862.289215] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4862.306176] Lustre: 167884:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4862.385064] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4870.053378] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 4870.241197] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4870.265149] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4870.564403] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4882.293779] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4887.817288] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4888.056255] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4893.567725] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4893.790576] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4896.233934] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4896.242766] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4896.251719] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4896.268200] Lustre: Skipped 1 previous similar message [ 4901.351074] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4901.354372] Lustre: Skipped 1 previous similar message [ 4902.120422] Lustre: server umount lustre-OST0000 complete [ 4906.463960] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4906.485910] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4906.495573] Lustre: Skipped 1 previous similar message [ 4911.910424] Lustre: server umount lustre-MDT0000 complete [ 4915.457765] LustreError: 165909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789639891 with bad export cookie 11330797652305005323 [ 4915.477904] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4915.931439] Lustre: server umount lustre-MDT0001 complete [ 4924.784968] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 06:11:40 (1789639900) [ 4945.554443] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [ 4958.166347] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 4996.047043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4996.349207] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4996.397757] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4996.493221] Lustre: lustre-MDT0000: new disk, initializing [ 4996.600899] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4996.625442] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5001.393419] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 5012.576624] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5012.798753] Lustre: 172642:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5012.906052] LustreError: 172653:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5013.291654] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5018.861715] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 5028.675943] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5034.719611] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5042.993376] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5043.446660] Lustre: lustre-OST0000: new disk, initializing [ 5043.452400] Lustre: Skipped 1 previous similar message [ 5043.458445] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5043.465912] Lustre: Skipped 2 previous similar messages [ 5043.476612] Lustre: 174619:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5051.015555] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 5051.995911] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5052.005764] Lustre: Skipped 1 previous similar message [ 5052.027635] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5052.166727] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5062.055417] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5072.872282] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5072.886669] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5072.892329] Lustre: Skipped 1 previous similar message [ 5072.901638] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5072.918651] LustreError: Skipped 1 previous similar message [ 5072.936345] Lustre: Skipped 3 previous similar messages [ 5084.127474] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5084.404934] Lustre: server umount lustre-OST0000 complete [ 5088.224316] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5088.234060] Lustre: Skipped 1 previous similar message [ 5093.348906] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5093.356569] Lustre: Skipped 7 previous similar messages [ 5094.058698] Lustre: server umount lustre-MDT0000 complete [ 5097.281825] LustreError: 172635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789640073 with bad export cookie 11330797652305006205 [ 5097.286489] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5097.292821] LustreError: 172635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 5097.671499] Lustre: server umount lustre-MDT0001 complete [ 5106.702703] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 06:14:41 (1789640081) [ 5122.370982] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [ 5132.551720] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 5168.446147] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5168.716249] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5168.760062] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5168.861759] Lustre: lustre-MDT0000: new disk, initializing [ 5169.018236] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5169.028144] Lustre: Skipped 2 previous similar messages [ 5169.054211] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5172.769106] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 5182.187739] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5182.329837] Lustre: 178942:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5182.651307] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5187.710389] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 5197.152276] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5202.947623] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5209.633839] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5209.857090] Lustre: 180918:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5215.289802] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5215.395920] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5215.676391] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 5223.783473] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5231.073046] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5231.094339] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5231.108678] Lustre: Skipped 1 previous similar message [ 5231.118200] LustreError: 181247:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5231.196947] Lustre: server umount lustre-OST0000 complete [ 5236.210280] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5236.216396] Lustre: Skipped 1 previous similar message [ 5240.856725] Lustre: server umount lustre-MDT0000 complete [ 5244.033946] LustreError: 178934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789640220 with bad export cookie 11330797652305007283 [ 5244.042733] LustreError: 178934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5244.045844] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5244.381646] Lustre: server umount lustre-MDT0001 complete [ 5253.709420] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 5256.129162] Key type lgssc unregistered [ 5256.440444] LNet: 182770:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5256.449604] LNetError: 182770:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5256.483856] LNet: Removed LNI 192.168.202.132@tcp [ 5257.317683] Key type .llcrypt unregistered [ 5257.319214] Key type ._llcrypt unregistered [ 5276.241331] Key type ._llcrypt registered [ 5276.243673] Key type .llcrypt registered [ 5276.325437] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 5277.442131] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5277.518516] alg: No test for adler32 (adler32-zlib) [ 5278.595264] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [ 5278.812368] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 5280.495204] Key type lgssc registered [ 5281.601644] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5288.781465] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5288.807093] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5290.260708] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5294.035280] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 5300.707576] LustreError: 184164:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5300.736326] LustreError: 184164:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5301.545405] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5301.915630] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5305.719116] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 5312.131393] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5317.406915] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5323.857043] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5324.379588] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5331.394526] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 5335.688193] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5341.229046] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5361.634767] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5361.645282] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5361.649930] Lustre: Skipped 1 previous similar message [ 5361.650406] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5361.655865] Lustre: Skipped 1 previous similar message [ 5364.887476] Lustre: server umount lustre-OST0000 complete [ 5371.877897] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5371.882776] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5371.901989] Lustre: Skipped 1 previous similar message [ 5371.914217] Lustre: Skipped 1 previous similar message [ 5374.840570] Lustre: server umount lustre-MDT0000 complete [ 5376.993617] LustreError: 184163:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5377.015163] LustreError: 184163:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5378.539500] LustreError: 184145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789640354 with bad export cookie 16112824282634030572 [ 5378.543146] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5378.545517] LustreError: 184145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5378.976667] Lustre: server umount lustre-MDT0001 complete [ 5390.231165] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 5392.917071] Key type lgssc unregistered [ 5393.212371] LNet: 187718:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5393.233244] LNetError: 187718:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5394.283714] LNet: Removed LNI 192.168.202.132@tcp [ 5395.180343] Key type .llcrypt unregistered [ 5395.182756] Key type ._llcrypt unregistered [ 5417.648176] Key type ._llcrypt registered [ 5417.652898] Key type .llcrypt registered [ 5417.756175] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 5418.655506] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5418.712890] alg: No test for adler32 (adler32-zlib) [ 5419.906512] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [ 5420.092792] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 5421.735255] Key type lgssc registered [ 5422.792897] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5429.862440] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5429.881972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5431.283746] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5435.204920] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 5441.517271] LustreError: 189133:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5441.532272] LustreError: 189133:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5444.296866] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5444.757368] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5448.984332] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 5456.619731] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5463.196983] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5471.710753] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5472.177494] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5476.283569] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5481.195935] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 5489.661721] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5496.801336] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5496.810131] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5496.821387] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5497.774239] Lustre: lustre-OST0000: Not available for connect from 192.168.202.32@tcp (stopping) [ 5499.874102] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5499.875834] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5499.897137] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5502.487663] Lustre: server umount lustre-OST0000 complete [ 5509.936243] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff99dec5b18800.ost_server_uuid 50 [ 5511.958538] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5525.105732] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5525.121650] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5525.130814] Lustre: Skipped 2 previous similar messages [ 5530.594346] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5530.605307] Lustre: Skipped 1 previous similar message [ 5531.395540] Lustre: server umount lustre-MDT0000 complete [ 5535.608661] LustreError: 189112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789640511 with bad export cookie 10966348604011588677 [ 5535.626927] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5535.632363] LustreError: 189112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5536.018857] Lustre: server umount lustre-MDT0001 complete [ 5545.259609] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 06:22:00 (1789640520) [ 5552.712701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5553.130885] LustreError: 193049:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5553.206453] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5557.903976] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 5558.244850] LustreError: 193050:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5563.385029] LustreError: 193049:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5566.328780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5566.697090] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5571.726572] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 5579.517595] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5585.756664] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5593.288939] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5593.801120] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5597.866030] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5600.468722] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 5609.634658] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5619.260507] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5619.467261] Lustre: lustre-OST0001: new disk, initializing [ 5619.471227] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5619.480823] Lustre: 195891:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5619.585208] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5625.507321] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5625.524536] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5625.670139] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5628.462122] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 5640.026801] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5644.256898] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5644.264325] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5644.277256] Lustre: Skipped 1 previous similar message [ 5644.285550] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5649.537493] Lustre: server umount lustre-OST0000 complete [ 5651.363899] LustreError: 195347:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5656.952150] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5657.287636] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5661.604831] LustreError: 194870:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5661.640599] LustreError: 194870:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 5663.478229] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5663.806718] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5682.149864] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5682.151934] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5682.168464] LustreError: 194872:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5682.168475] LustreError: 194872:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 9 previous similar messages [ 5682.173934] LustreError: Skipped 1 previous similar message [ 5682.186881] Lustre: Skipped 1 previous similar message [ 5682.191510] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5682.191519] Lustre: Skipped 3 previous similar messages [ 5684.802974] Lustre: server umount lustre-OST0001 complete [ 5689.825587] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5692.389500] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5692.430441] Lustre: Skipped 2 previous similar messages [ 5695.941682] Lustre: server umount lustre-MDT0000 complete [ 5700.075498] LustreError: 193029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789640676 with bad export cookie 10966348604011589748 [ 5700.089829] LustreError: 193029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5700.091609] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5700.462313] Lustre: server umount lustre-MDT0001 complete [ 5726.085481] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 06:25:00 (1789640700) [ 5735.315814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5735.811618] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5735.886211] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5736.203643] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5741.818288] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 5752.121699] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5752.195126] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5752.214751] Lustre: 199330:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5757.515668] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 5765.693596] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5771.785311] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5779.440569] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5779.703238] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5785.722217] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 5787.305531] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 5792.401575] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5799.559100] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5799.617747] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5806.396087] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 5807.525667] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 5817.201084] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5821.925254] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5821.939078] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5821.943226] Lustre: Skipped 4 previous similar messages [ 5826.215452] Lustre: server umount lustre-OST0000 complete [ 5827.047873] LustreError: 201640:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5827.069160] LustreError: 201640:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 5842.405250] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5842.417977] LustreError: Skipped 1 previous similar message [ 5847.685936] Lustre: server umount lustre-OST0001 complete [ 5857.629425] Lustre: server umount lustre-MDT0000 complete [ 5861.276192] LustreError: 199325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789640837 with bad export cookie 10966348604011591155 [ 5861.278364] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5861.284809] LustreError: 199325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5861.654544] Lustre: server umount lustre-MDT0001 complete [ 5884.323966] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 06:27:39 (1789640859) [ 5902.411254] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [ 5913.755684] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 5950.369263] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5950.602862] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5950.620821] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5950.686464] Lustre: lustre-MDT0000: new disk, initializing [ 5950.751953] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5950.757745] Lustre: Skipped 3 previous similar messages [ 5950.772759] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5955.183233] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 5965.972122] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5966.074963] Lustre: 207430:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5966.134881] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5966.139829] Lustre: Skipped 1 previous similar message [ 5966.213929] Lustre: lustre-MDT0001: new disk, initializing [ 5966.374916] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5966.394199] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5971.033490] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 5981.065708] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5986.433288] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5993.793745] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5994.127089] Lustre: lustre-OST0000: new disk, initializing [ 5994.131932] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5994.145939] Lustre: 209401:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6000.661703] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6003.883031] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6003.897623] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6004.009089] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6010.290833] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6016.241828] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6016.510757] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6021.682001] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6021.920096] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6024.673225] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6024.682116] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6024.685564] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6024.697169] Lustre: Skipped 6 previous similar messages [ 6024.707260] Lustre: Skipped 7 previous similar messages [ 6030.717671] Lustre: server umount lustre-OST0000 complete [ 6042.820182] Lustre: server umount lustre-MDT0000 complete [ 6045.158364] LustreError: 207436:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6045.188357] LustreError: 207436:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 14 previous similar messages [ 6046.913781] LustreError: 208225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789641023 with bad export cookie 10966348604011592940 [ 6046.914666] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6046.922755] LustreError: 208225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6047.540907] Lustre: server umount lustre-MDT0001 complete [ 6056.312167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6061.059440] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6069.544813] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6074.705246] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6083.238993] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6088.490500] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6095.501356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6095.850765] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6095.855450] Lustre: Skipped 4 previous similar messages [ 6103.114378] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6111.635239] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6116.418587] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6116.658272] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6122.163834] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6122.373656] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6125.536288] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6125.545145] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6125.546849] LustreError: Skipped 1 previous similar message [ 6125.570332] Lustre: Skipped 3 previous similar messages [ 6130.623819] Lustre: server umount lustre-OST0000 complete [ 6137.216688] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6142.119281] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6171.469118] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6178.478643] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6185.972403] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6189.826438] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 2 sec [ 6190.439155] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6190.463507] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6190.464489] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6190.498340] Lustre: Skipped 1 previous similar message [ 6195.680028] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6195.684755] LustreError: Skipped 1 previous similar message [ 6195.687351] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6195.698312] Lustre: Skipped 10 previous similar messages [ 6196.213971] Lustre: server umount lustre-OST0000 complete [ 6214.117786] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6214.438565] Lustre: server umount lustre-MDT0000 complete [ 6216.171671] LustreError: 211620:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6216.190865] LustreError: 211620:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 6217.920797] LustreError: 215727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789641194 with bad export cookie 10966348604011593822 [ 6217.923360] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6217.930451] LustreError: 215727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6218.309459] Lustre: server umount lustre-MDT0001 complete [ 6226.542304] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 06:33:21 (1789641201) [ 6233.734205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6239.100285] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6248.176066] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6253.504508] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6261.265108] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6268.177769] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6277.045925] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6286.635211] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6294.603782] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6299.242527] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6299.528524] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6304.267202] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6304.551926] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6311.883992] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6312.100528] Lustre: lustre-OST0001: new disk, initializing [ 6312.105375] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6312.110569] Lustre: 220954:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6318.730464] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6318.744470] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6318.927987] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6319.498176] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6328.802208] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6333.670180] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6333.921922] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6338.593757] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6338.815081] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6344.672828] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6344.681472] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6344.697259] Lustre: Skipped 5 previous similar messages [ 6346.915525] Lustre: server umount lustre-OST0001 complete [ 6353.266458] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6353.528654] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6357.986652] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6358.280300] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6387.055652] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6387.231857] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6387.237659] Lustre: Skipped 5 previous similar messages [ 6392.308507] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6392.327810] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6392.330395] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6393.655872] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6401.336223] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6402.930563] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6410.911955] Lustre: server umount lustre-OST0001 complete [ 6435.718936] Lustre: server umount lustre-OST0000 complete [ 6445.815614] Lustre: server umount lustre-MDT0000 complete [ 6449.885499] LustreError: 217567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789641426 with bad export cookie 10966348604011594788 [ 6449.889531] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6449.892611] LustreError: 217567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6456.399894] Lustre: server umount lustre-MDT0001 complete [ 6482.027172] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 06:37:37 (1789641457) [ 6488.448989] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6488.624733] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6488.643945] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6494.417690] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6503.175473] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6503.276823] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6503.326938] Lustre: 226198:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6508.375590] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6516.200948] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6521.940839] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6528.067744] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6528.255096] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6534.026814] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6540.647832] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6550.903750] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6551.102702] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6560.133761] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6569.691287] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6572.019925] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6577.975991] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6578.280204] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6584.819713] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6585.008511] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6592.701579] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff99dec58ce800.ost_server_uuid 50 [ 6594.333380] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff99dec58ce800.ost_server_uuid in IDLE state after 0 sec [ 6595.453640] Lustre: Permanently deactivating lustre-OST0001 [ 6595.467979] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6599.518272] Lustre: Permanently reactivating lustre-OST0001 [ 6599.538477] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6599.542386] Lustre: Skipped 2 previous similar messages [ 6606.306136] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6606.318122] LustreError: Skipped 5 previous similar messages [ 6606.320977] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6606.336997] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6606.342704] Lustre: Skipped 7 previous similar messages [ 6606.346108] Lustre: Skipped 18 previous similar messages [ 6606.835770] Lustre: server umount lustre-OST0001 complete [ 6611.445765] LustreError: 228042:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6611.490820] LustreError: 228042:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 39 previous similar messages [ 6617.772798] Lustre: server umount lustre-OST0000 complete [ 6629.029509] Lustre: server umount lustre-MDT0000 complete [ 6634.323405] LustreError: 228059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789641610 with bad export cookie 10966348604011596076 [ 6634.326686] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6634.332879] LustreError: 228059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6635.270356] Lustre: server umount lustre-MDT0001 complete [ 6664.296646] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 06:40:39 (1789641639) [ 6672.038579] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6672.360727] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6672.394409] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6672.402530] Lustre: Skipped 2 previous similar messages [ 6677.417788] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6686.000554] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6686.096286] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6686.125302] Lustre: 232897:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6691.837796] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6701.512071] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6708.311032] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6716.782855] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6717.070943] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6717.118429] Lustre: Permanently deactivating lustre-OST0000 [ 6717.132240] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6725.537493] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6725.655792] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6725.667818] LustreError: 235129:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0001: can't precreate: rc = -5 [ 6725.673457] LustreError: 235129:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0001: cannot precreate objects: rc = -5 [ 6735.563072] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6744.679501] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6744.883379] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6755.204953] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6755.614071] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6765.870877] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6768.136086] Lustre: Permanently reactivating lustre-OST0000 [ 6768.172105] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6768.178566] Lustre: Skipped 2 previous similar messages [ 6778.521694] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6778.537164] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6778.566472] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6778.590452] Lustre: Skipped 1 previous similar message [ 6778.669332] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 6778.721103] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6788.902309] Lustre: server umount lustre-OST0001 complete [ 6799.826243] Lustre: server umount lustre-OST0000 complete [ 6812.024208] Lustre: server umount lustre-MDT0000 complete [ 6817.576758] LustreError: 234766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789641793 with bad export cookie 10966348604011597756 [ 6817.591701] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6817.598537] LustreError: 234766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6818.104027] Lustre: server umount lustre-MDT0001 complete [ 6833.790672] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 6838.337881] Key type lgssc unregistered [ 6838.690496] LNet: 237781:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6838.702079] LNetError: 237781:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6838.727497] LNet: Removed LNI 192.168.202.132@tcp [ 6839.979960] Key type .llcrypt unregistered [ 6839.982400] Key type ._llcrypt unregistered [ 6867.077561] Key type ._llcrypt registered [ 6867.080655] Key type .llcrypt registered [ 6867.227359] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 6868.445960] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6868.518845] alg: No test for adler32 (adler32-zlib) [ 6869.674575] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [ 6870.033074] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 6871.743967] Key type lgssc registered [ 6872.914057] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6883.121793] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6883.136901] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6884.528482] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6890.115331] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6894.057921] LustreError: 239244:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6894.099224] LustreError: 239244:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6899.186211] LustreError: 239245:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6900.036265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6900.231774] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6900.260849] Lustre: 239232:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6900.284318] Lustre: Permanently deactivating lustre-MDT0001 [ 6900.314699] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6900.894533] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6906.947326] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6909.544616] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6916.634366] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6923.897132] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6935.473513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6936.226228] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6943.932130] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6945.402079] LustreError: 241087:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6950.893503] LustreError: 241089:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6952.742898] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6954.013077] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 6959.078237] LustreError: 241519:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6959.115856] LustreError: 241519:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 6960.197171] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6960.630673] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6965.760313] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 6965.786166] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 6967.769712] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6975.748516] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6978.888774] Lustre: Permanently reactivating lustre-MDT0001 [ 6978.917482] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 6978.926257] Lustre: Skipped 2 previous similar messages [ 6981.661122] Lustre: lustre-MDT0001: Client 8aedc546-484f-4305-9f96-c09f15edda8a (at 192.168.202.32@tcp) reconnecting [ 6989.107715] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6989.496373] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6989.516318] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 6989.533450] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 6989.561782] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6989.561148] LustreError: 243008:0:(llog_osd.c:2268:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 6989.577986] LustreError: 243008:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 6989.584556] LustreError: 243008:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.132@tcp: cfg command failed: rc = -108 [ 6989.591378] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 6989.591378] [ 6989.596830] LustreError: 239236:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 6990.991605] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6997.744735] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6997.976894] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7003.315903] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7003.448835] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7006.216639] Lustre: Permanently deactivating lustre-MDT0001 [ 7006.236973] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7006.242807] Lustre: Skipped 2 previous similar messages [ 7011.957385] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7020.154340] Lustre: server umount lustre-MDT0000 complete [ 7022.064067] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7022.080696] LustreError: 239990:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7022.093297] Lustre: Skipped 3 previous similar messages [ 7022.128389] LustreError: 239990:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 7024.391217] LustreError: 242795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789642000 with bad export cookie 18002302483443558661 [ 7024.403958] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7024.404982] LustreError: 242795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7024.983947] Lustre: server umount lustre-MDT0001 complete [ 7029.656723] Lustre: server umount lustre-OST0000 complete [ 7042.065374] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 06:46:57 (1789642017) [ 7053.715520] Lustre: server umount lustre-OST0001 complete [ 7061.225142] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [ 7072.643177] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 7110.594452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7110.956695] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7110.965629] Lustre: Skipped 2 previous similar messages [ 7110.985208] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7111.093718] Lustre: lustre-MDT0000: new disk, initializing [ 7111.236593] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7111.282995] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7116.427467] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7127.653186] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7127.770564] Lustre: 247506:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7127.829780] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7127.833484] Lustre: Skipped 1 previous similar message [ 7127.901503] Lustre: lustre-MDT0001: new disk, initializing [ 7127.989185] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7128.020135] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7128.037775] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7132.555164] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7142.270450] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7148.042456] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7154.853976] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7155.097272] Lustre: lustre-OST0000: new disk, initializing [ 7155.100771] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7155.106496] Lustre: 249474:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7155.193156] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7160.376069] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7160.385786] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7160.537792] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7162.077475] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7173.054658] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7175.319980] LustreError: 247514:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7185.411424] LustreError: 247514:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7185.458531] LustreError: 247513:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7185.656463] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7185.857411] Lustre: lustre-OST0001: new disk, initializing [ 7185.863100] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7185.873536] Lustre: 250588:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7194.135481] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7194.730951] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7194.746548] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7194.893446] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7195.537393] LustreError: 247513:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7206.404996] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7208.497535] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7215.590219] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7215.593321] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7215.594924] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7215.611174] Lustre: Skipped 1 previous similar message [ 7215.629501] LustreError: Skipped 1 previous similar message [ 7216.846178] Lustre: server umount lustre-OST0001 complete [ 7220.718982] LustreError: 249458:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7220.746386] LustreError: 249458:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7236.065402] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7236.071485] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7236.097172] Lustre: Skipped 1 previous similar message [ 7236.124910] Lustre: Skipped 2 previous similar messages [ 7241.201782] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7241.214129] Lustre: Skipped 1 previous similar message [ 7246.315206] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7246.338085] Lustre: Skipped 1 previous similar message [ 7249.375360] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7249.783505] Lustre: server umount lustre-OST0000 complete [ 7256.033343] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7256.048856] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7256.062677] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7266.784280] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7266.793827] Lustre: Skipped 3 previous similar messages [ 7269.343150] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7269.517676] Lustre: server umount lustre-MDT0000 complete [ 7271.909685] LustreError: 250144:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7271.927649] LustreError: 250144:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [ 7273.354703] LustreError: 248300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789642249 with bad export cookie 18002302483443561258 [ 7273.364214] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7273.371021] LustreError: 248300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7280.023607] Lustre: server umount lustre-MDT0001 complete [ 7293.174576] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 7296.193753] Key type lgssc unregistered [ 7296.451969] LNet: 252628:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7296.461402] LNetError: 252628:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7296.478530] LNet: Removed LNI 192.168.202.132@tcp [ 7297.381220] Key type .llcrypt unregistered [ 7297.384184] Key type ._llcrypt unregistered [ 7342.361985] Key type ._llcrypt registered [ 7342.373573] Key type .llcrypt registered [ 7342.563135] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 7343.837357] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7343.880654] alg: No test for adler32 (adler32-zlib) [ 7344.954286] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [ 7345.268074] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 7346.927261] Key type lgssc registered [ 7348.135784] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7356.133652] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7356.150718] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7357.444556] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7357.469484] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7357.664572] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7362.837578] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7371.888239] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7371.998593] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7372.035944] Lustre: 254836:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7372.405844] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7377.701740] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7386.239277] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7391.859186] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7398.358754] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7398.586648] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7398.754260] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7404.933974] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7405.304340] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7412.270816] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7466.466674] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7466.472820] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7466.485212] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7466.977042] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7467.005540] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7468.282972] Lustre: server umount lustre-OST0000 complete [ 7472.835813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7488.074565] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7488.589153] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7495.999228] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7504.229072] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7506.950221] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7506.969922] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7506.982841] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7507.001345] Lustre: Skipped 1 previous similar message [ 7547.873434] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7547.878532] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7547.919823] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7550.363494] Lustre: server umount lustre-OST0000 complete [ 7556.576309] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7556.590512] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7556.601822] Lustre: Skipped 1 previous similar message [ 7556.611080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7556.617140] Lustre: Skipped 1 previous similar message [ 7560.721551] Lustre: server umount lustre-MDT0000 complete [ 7565.594906] LustreError: 254827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789642541 with bad export cookie 5623448612115325034 [ 7565.597375] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7565.617182] LustreError: 254827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7566.276665] Lustre: server umount lustre-MDT0001 complete [ 7577.818750] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 7580.742780] Key type lgssc unregistered [ 7581.138910] LNet: 260258:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7581.159718] LNetError: 260258:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7581.190698] LNet: Removed LNI 192.168.202.132@tcp [ 7582.068789] Key type .llcrypt unregistered [ 7582.072671] Key type ._llcrypt unregistered [ 7605.535384] Key type ._llcrypt registered [ 7605.537574] Key type .llcrypt registered [ 7605.627892] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 7606.847503] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7606.906968] alg: No test for adler32 (adler32-zlib) [ 7607.943945] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [ 7608.152775] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 7609.799240] Key type lgssc registered [ 7610.774576] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7618.358149] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7618.373968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7619.721124] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7623.440721] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7629.300978] LustreError: 261686:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7629.319732] LustreError: 261686:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7630.132934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7630.447840] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7634.403633] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7641.471743] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7647.011817] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7653.246651] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7653.625899] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7660.517187] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7663.856654] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7669.504592] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7691.750986] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7691.761594] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7691.778518] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7694.820533] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7694.828506] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7694.837059] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7697.586539] Lustre: server umount lustre-OST0000 complete [ 7701.985701] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7701.992451] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7702.016928] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7702.025758] Lustre: Skipped 1 previous similar message [ 7705.056825] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7705.057386] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7707.674878] Lustre: server umount lustre-MDT0000 complete [ 7710.177755] LustreError: 261683:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7710.195561] LustreError: 261683:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7711.326368] LustreError: 261667:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789642687 with bad export cookie 17435969760591573786 [ 7711.332134] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7711.335872] LustreError: 261667:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7711.617513] Lustre: server umount lustre-MDT0001 complete [ 7721.536153] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 7724.288649] Key type lgssc unregistered [ 7724.591541] LNet: 265986:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7724.597913] LNetError: 265986:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7724.622742] LNet: Removed LNI 192.168.202.132@tcp [ 7725.550899] Key type .llcrypt unregistered [ 7725.553993] Key type ._llcrypt unregistered [ 7746.583270] Key type ._llcrypt registered [ 7746.587551] Key type .llcrypt registered [ 7746.721516] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 7747.919599] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7748.027820] alg: No test for adler32 (adler32-zlib) [ 7749.328726] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [ 7749.582960] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 7751.239372] Key type lgssc registered [ 7752.788801] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7760.680995] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7760.721993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7762.237701] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7767.418156] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7772.645912] LustreError: 267204:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7772.660672] LustreError: 267204:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7775.636590] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7776.098443] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7781.296599] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7789.579250] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7795.666848] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7801.713722] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7802.098931] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7807.218201] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7807.707158] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7815.386235] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7832.037845] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7832.042505] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7832.047795] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7832.089085] Lustre: Skipped 1 previous similar message [ 7837.154027] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7837.166222] Lustre: Skipped 2 previous similar messages [ 7838.225658] Lustre: server umount lustre-OST0000 complete [ 7847.418388] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7847.433166] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7847.461492] Lustre: Skipped 1 previous similar message [ 7849.630986] Lustre: server umount lustre-MDT0000 complete [ 7852.515722] LustreError: 267932:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7852.536218] LustreError: 267932:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7853.919962] LustreError: 267186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789642830 with bad export cookie 8921688426267983866 [ 7853.932870] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7853.941711] LustreError: 267186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7854.255336] Lustre: server umount lustre-MDT0001 complete [ 7868.967982] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 7872.682477] Key type lgssc unregistered [ 7872.953485] LNet: 270880:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7872.977659] LNetError: 270880:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7872.993285] LNet: Removed LNI 192.168.202.132@tcp [ 7873.839945] Key type .llcrypt unregistered [ 7873.841717] Key type ._llcrypt unregistered [ 7898.506190] Key type ._llcrypt registered [ 7898.513218] Key type .llcrypt registered [ 7898.641853] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 7899.867781] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7900.095375] alg: No test for adler32 (adler32-zlib) [ 7901.342878] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [ 7901.792899] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 7903.463267] Key type lgssc registered [ 7905.080991] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7914.379618] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7914.399289] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7916.011353] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7920.608239] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7926.243479] LustreError: 272307:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7926.271907] LustreError: 272307:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7929.858675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7930.425962] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7935.841600] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7944.390871] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7950.714093] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7957.101517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7957.349765] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7961.472761] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7964.343707] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7971.712862] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8000.480297] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8000.489845] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8000.525294] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8002.530450] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8002.538342] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8002.556266] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8003.560172] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8003.565209] Lustre: Skipped 1 previous similar message [ 8005.627697] Lustre: server umount lustre-OST0000 complete [ 8012.307319] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8012.319823] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8012.389782] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8015.779856] Lustre: server umount lustre-MDT0000 complete [ 8018.913238] LustreError: 272303:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8018.946058] LustreError: 272303:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8020.552665] LustreError: 272289:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789642996 with bad export cookie 3774726680161646074 [ 8020.564775] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8020.567259] LustreError: 272289:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8020.952243] Lustre: server umount lustre-MDT0001 complete [ 8034.218121] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 8037.901130] Key type lgssc unregistered [ 8038.174938] LNet: 276613:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8038.191937] LNetError: 276613:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8038.234709] LNet: Removed LNI 192.168.202.132@tcp [ 8038.994993] Key type .llcrypt unregistered [ 8038.996791] Key type ._llcrypt unregistered [ 8057.214193] Key type ._llcrypt registered [ 8057.217909] Key type .llcrypt registered [ 8057.351207] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 8058.474628] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8058.655545] alg: No test for adler32 (adler32-zlib) [ 8059.674757] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [ 8059.934945] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 8061.623893] Key type lgssc registered [ 8062.973398] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8071.308598] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8071.332489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8072.889256] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8077.652507] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8083.426644] LustreError: 277846:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8083.443423] LustreError: 277846:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8087.324742] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8087.770808] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8093.087817] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8102.235168] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8108.321548] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8114.363069] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8114.819667] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8117.947357] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8120.888614] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8128.723991] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8143.840835] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8143.851575] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8143.870267] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8147.940149] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8147.943500] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8149.387868] Lustre: server umount lustre-OST0000 complete [ 8154.597853] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8154.605554] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8154.621202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8154.624128] Lustre: Skipped 1 previous similar message [ 8158.178663] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8158.180926] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8158.217763] Lustre: Skipped 1 previous similar message [ 8159.947829] Lustre: server umount lustre-MDT0000 complete [ 8163.297751] LustreError: 277854:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8163.312507] LustreError: 277854:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8163.871131] LustreError: 277813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789643140 with bad export cookie 1887034531993638806 [ 8163.876338] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8163.896936] LustreError: 277813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8164.466850] Lustre: server umount lustre-MDT0001 complete [ 8175.955237] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 8179.132952] Key type lgssc unregistered [ 8179.464970] LNet: 281546:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8179.475403] LNetError: 281546:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8179.493693] LNet: Removed LNI 192.168.202.132@tcp [ 8180.281332] Key type .llcrypt unregistered [ 8180.283498] Key type ._llcrypt unregistered [ 8192.151968] Key type ._llcrypt registered [ 8192.153740] Key type .llcrypt registered [ 8192.249310] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8211.152323] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [ 8225.715814] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 8226.904294] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8227.212934] alg: No test for adler32 (adler32-zlib) [ 8228.292394] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [ 8228.483472] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 8230.183323] Key type lgssc registered [ 8231.310607] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8268.628415] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8268.678570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8269.931075] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8269.965877] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8270.023822] Lustre: lustre-MDT0000: new disk, initializing [ 8270.129429] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8270.153596] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8276.041790] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8290.658396] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8290.885733] Lustre: 285100:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8290.995750] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8291.001867] Lustre: Skipped 1 previous similar message [ 8291.124592] Lustre: lustre-MDT0001: new disk, initializing [ 8291.272735] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8291.311945] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8291.326166] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8296.120335] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8305.765691] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8311.510243] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8319.227467] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8319.540881] Lustre: lustre-OST0000: new disk, initializing [ 8319.544288] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8319.550923] Lustre: 287069:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8319.625750] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8326.806938] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8327.735759] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8327.747231] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8327.894612] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8336.304454] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8341.931992] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8342.319568] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8347.151364] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8347.477873] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8353.760743] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8353.765769] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8353.772532] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8353.802480] Lustre: Skipped 1 previous similar message [ 8355.784691] Lustre: server umount lustre-OST0000 complete [ 8362.977659] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8362.990195] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8363.027066] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8363.033486] Lustre: Skipped 1 previous similar message [ 8364.005876] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8364.028692] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8364.039264] Lustre: Skipped 1 previous similar message [ 8366.086753] Lustre: server umount lustre-MDT0000 complete [ 8369.463979] LustreError: 285896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789643345 with bad export cookie 5016206590909278072 [ 8369.467501] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8369.479361] LustreError: 285896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8370.184915] Lustre: server umount lustre-MDT0001 complete [ 8378.165363] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 07:09:13 (1789643353) [ 8385.514774] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8385.983333] LustreError: 289481:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8386.003554] LustreError: 289481:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 8386.052746] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8391.013620] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8391.142835] LustreError: 289482:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8391.160860] LustreError: 289482:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8396.260270] LustreError: 289481:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8400.946809] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8401.447308] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8406.276354] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8413.471963] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8419.193848] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8426.188958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8426.699222] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8433.211893] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8441.070744] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8461.795277] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8461.798544] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8461.799344] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8461.812901] Lustre: Skipped 1 previous similar message [ 8463.880546] Lustre: server umount lustre-OST0000 complete [ 8467.937496] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8467.950065] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8467.975686] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8467.985268] Lustre: Skipped 1 previous similar message [ 8474.040338] Lustre: server umount lustre-MDT0000 complete [ 8477.163978] LustreError: 290473:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8477.202521] LustreError: 290473:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8477.610047] LustreError: 289462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789643453 with bad export cookie 5016206590909278954 [ 8477.631543] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8478.233367] Lustre: server umount lustre-MDT0001 complete [ 8488.664448] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 8491.487958] Key type lgssc unregistered [ 8491.756473] LNet: 293036:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8491.766342] LNetError: 293036:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8491.785803] LNet: Removed LNI 192.168.202.132@tcp [ 8492.889223] Key type .llcrypt unregistered [ 8492.891745] Key type ._llcrypt unregistered [ 8522.619417] Key type ._llcrypt registered [ 8522.621627] Key type .llcrypt registered [ 8522.714961] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [ 8538.584647] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 8539.944264] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8540.083754] alg: No test for adler32 (adler32-zlib) [ 8541.239281] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [ 8541.526829] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 8543.191385] Key type lgssc registered [ 8544.430231] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8591.753698] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8591.772039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8593.045195] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8593.068728] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8593.136186] Lustre: lustre-MDT0000: new disk, initializing [ 8593.204155] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8593.221183] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8597.545835] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8607.609589] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8607.769248] Lustre: 296861:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8607.858410] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8607.863301] Lustre: Skipped 1 previous similar message [ 8607.982763] Lustre: lustre-MDT0001: new disk, initializing [ 8608.130446] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8608.172767] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8608.190257] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8612.774311] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8621.549136] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8626.982654] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8634.989582] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8635.405342] Lustre: lustre-OST2710: new disk, initializing [ 8635.409798] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8635.414586] Lustre: 298829:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8635.554822] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8643.635686] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8645.298951] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8645.323591] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8645.606686] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8656.659828] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8664.494425] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8664.725651] Lustre: lustre-OST03e8: new disk, initializing [ 8664.736879] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8664.743844] Lustre: 299891:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8664.835049] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8670.318183] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8670.331578] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8670.431408] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8672.635336] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8682.704880] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8690.749864] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8691.127645] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8696.600373] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8696.943877] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8704.685276] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8704.959418] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8711.344442] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8711.585361] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8715.744980] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8715.748322] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8715.759806] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8716.770912] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8716.771734] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8716.777910] Lustre: Skipped 2 previous similar messages [ 8716.782992] Lustre: Skipped 2 previous similar messages [ 8722.194211] Lustre: server umount lustre-MDT0000 complete [ 8726.563965] LustreError: 296853:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789643702 with bad export cookie 9838023097781727096 [ 8726.578915] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8727.098485] Lustre: server umount lustre-MDT0001 complete [ 8741.886457] Lustre: server umount lustre-OST2710 complete [ 8743.199214] Lustre: 294656:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789643703/real 1789643703] req@00000000288b53b6 x1876577240555136/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789643719 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8743.247257] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8746.907435] Lustre: server umount lustre-OST03e8 complete [ 8765.081188] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [ 8774.714514] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 8808.949447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8809.198524] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8809.220700] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8809.294777] Lustre: lustre-MDT0000: new disk, initializing [ 8809.364313] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8809.374587] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8814.074834] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8823.946594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8824.005602] Lustre: 305143:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8824.045971] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8824.049159] Lustre: Skipped 1 previous similar message [ 8824.103992] Lustre: lustre-MDT0001: new disk, initializing [ 8824.186685] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8824.218288] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8824.237609] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8828.745551] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8837.044392] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8842.295799] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8847.989302] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8848.258287] Lustre: lustre-OST0000: new disk, initializing [ 8848.262822] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8848.270492] Lustre: 307107:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8848.405360] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8853.596882] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8853.604494] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8853.683041] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8854.641736] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8863.487316] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8868.067467] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8868.278653] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8873.148072] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8873.349386] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8879.599199] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8879.602979] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8879.609072] LustreError: Skipped 1 previous similar message [ 8879.635214] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8879.637284] Lustre: Skipped 1 previous similar message [ 8879.665420] Lustre: Skipped 2 previous similar messages [ 8884.705030] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8884.708653] Lustre: Skipped 1 previous similar message [ 8889.311596] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8889.717937] Lustre: server umount lustre-OST0000 complete [ 8894.945268] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8894.951446] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8894.968359] Lustre: Skipped 1 previous similar message [ 8894.986308] Lustre: Skipped 1 previous similar message [ 8900.908867] Lustre: server umount lustre-MDT0000 complete [ 8904.956159] LustreError: 305134:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789643881 with bad export cookie 9838023097781728776 [ 8904.959848] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8904.983047] LustreError: 305134:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 8905.344236] Lustre: server umount lustre-MDT0001 complete [ 8915.706204] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 07:18:10 (1789643890) [ 8917.526161] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8920.024873] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 07:18:14 (1789643894) [ 8924.933564] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 8956.265451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8956.508809] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8956.539838] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8956.766650] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8962.845533] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8972.198186] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8972.373847] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8972.453133] Lustre: 310973:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8978.026617] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 9001.131023] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9001.394349] Lustre: Denying initial registration attempt from nid 192.168.202.132@tcp, specified as failover [ 9001.411117] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9001.424438] LustreError: 312428:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -99 [ 9001.432338] LustreError: 312428:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 9001.436272] LustreError: 312428:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9001.516858] Lustre: server umount lustre-OST0000 complete [ 9001.521902] LustreError: 312428:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9008.608513] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9008.622074] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9008.641555] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9008.646534] Lustre: Skipped 2 previous similar messages [ 9014.060932] Lustre: server umount lustre-MDT0000 complete [ 9017.700220] LustreError: 310963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789643993 with bad export cookie 9838023097781729651 [ 9017.707611] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9017.714672] LustreError: 310963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9018.004231] Lustre: server umount lustre-MDT0001 complete [ 9030.687100] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 9033.566507] Key type lgssc unregistered [ 9033.826053] LNet: 313546:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9033.834860] LNetError: 313546:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9034.853695] LNet: Removed LNI 192.168.202.132@tcp [ 9036.091358] Key type .llcrypt unregistered [ 9036.092807] Key type ._llcrypt unregistered [ 9047.388608] Key type ._llcrypt registered [ 9047.390496] Key type .llcrypt registered [ 9047.485559] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 9048.444302] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9048.525537] alg: No test for adler32 (adler32-zlib) [ 9049.563363] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [ 9049.743033] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 9051.383278] Key type lgssc registered [ 9052.521168] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9098.684885] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 9110.774968] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9110.799672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9112.151983] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9112.202398] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9112.464264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9117.304372] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 9127.759434] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9127.993739] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9128.049969] Lustre: 316311:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9128.363388] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 9128.476598] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9134.200122] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 9142.417499] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9148.977157] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9168.942612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9169.282292] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9169.478043] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9175.325566] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 9183.571135] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9187.296454] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9187.314682] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9187.332364] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9187.811354] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9187.826915] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9191.881308] Lustre: server umount lustre-OST0000 complete [ 9198.054249] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9198.067493] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9198.075374] Lustre: Skipped 1 previous similar message [ 9198.094686] Lustre: Skipped 2 previous similar messages [ 9203.174140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9203.183327] Lustre: Skipped 1 previous similar message [ 9208.315794] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9210.335803] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9210.585944] Lustre: server umount lustre-MDT0000 complete [ 9214.367810] LustreError: 316301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789644190 with bad export cookie 4167073278313717378 [ 9214.380049] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9214.609567] Lustre: server umount lustre-MDT0001 complete [ 9225.227737] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 9228.310824] Key type lgssc unregistered [ 9228.700501] LNet: 319885:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9228.715670] LNetError: 319885:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9229.750259] LNet: Removed LNI 192.168.202.132@tcp [ 9230.654809] Key type .llcrypt unregistered [ 9230.656807] Key type ._llcrypt unregistered [ 9258.855810] Key type ._llcrypt registered [ 9258.857430] Key type .llcrypt registered [ 9259.012486] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 9260.264966] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9260.293152] alg: No test for adler32 (adler32-zlib) [ 9261.381580] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [ 9261.625397] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 9263.311412] Key type lgssc registered [ 9264.253959] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9272.325467] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9272.362876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9273.944250] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9278.900102] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 9284.069615] LustreError: 321301:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9284.097573] LustreError: 321301:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9287.363707] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9287.818469] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9292.894639] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 9300.624842] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9306.806509] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9315.820150] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9316.332907] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9323.570232] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 9332.673165] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9343.970611] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9343.972580] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9343.989670] Lustre: Skipped 1 previous similar message [ 9345.698138] Lustre: server umount lustre-MDT0000 complete [ 9349.101795] LustreError: 321300:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9349.129034] LustreError: 321300:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9350.723407] LustreError: 321281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789644327 with bad export cookie 10433254380254657048 [ 9350.729411] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9350.747467] LustreError: 321281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9354.209342] LustreError: 323143:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9354.212426] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9354.233870] LustreError: 323143:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9354.271271] Lustre: Skipped 1 previous similar message [ 9354.284602] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9354.290944] Lustre: Skipped 3 previous similar messages [ 9357.319352] Lustre: server umount lustre-MDT0001 complete [ 9364.553659] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9374.757603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9375.967359] LustreError: 324881:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 9375.982376] LustreError: 324881:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@00000000acf652e3 x1876577995499136/t0(0) o250->MGC192.168.202.132@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1789644352 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9376.005834] LustreError: 324881:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 9376.692385] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x90ca5d94c25e1848 [ 9376.706386] Lustre: MGC192.168.202.132@tcp: Connection restored to 0@lo (at 0@lo) [ 9377.159563] LustreError: 323133:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9377.184813] LustreError: 323133:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9377.342161] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9377.434280] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9383.999084] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 9385.442437] LustreError: 324896:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9385.459169] LustreError: 324896:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9386.491584] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9392.844646] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9393.141915] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9398.064204] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 9398.250559] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9407.048366] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9408.641250] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9415.232754] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9417.075459] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9423.851092] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9423.871529] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9423.881497] Lustre: Skipped 2 previous similar messages [ 9428.451172] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9428.457301] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9428.470879] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9434.084285] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9434.097112] Lustre: Skipped 2 previous similar messages [ 9438.175402] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9438.425228] Lustre: server umount lustre-OST0000 complete [ 9444.350553] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9444.353096] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9444.374379] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9444.416051] Lustre: Skipped 1 previous similar message [ 9449.411404] Lustre: server umount lustre-MDT0000 complete [ 9449.452417] LustreError: 324900:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9449.482030] LustreError: 324900:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 9453.363586] LustreError: 323884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789644429 with bad export cookie 10433254380254664776 [ 9453.367399] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9453.375319] LustreError: 323884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9454.006762] Lustre: server umount lustre-MDT0001 complete [ 9466.296731] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 9469.207602] Key type lgssc unregistered [ 9469.569666] LNet: 327570:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9469.593594] LNetError: 327570:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9470.631226] LNet: Removed LNI 192.168.202.132@tcp [ 9471.695183] Key type .llcrypt unregistered [ 9471.701233] Key type ._llcrypt unregistered [ 9498.154488] Key type ._llcrypt registered [ 9498.156521] Key type .llcrypt registered [ 9498.239308] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 9499.271780] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9499.320400] alg: No test for adler32 (adler32-zlib) [ 9500.352771] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [ 9500.449433] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 9502.047216] Key type lgssc registered [ 9503.142818] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9510.961532] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9510.975163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9512.505483] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9517.315558] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 9522.659493] LustreError: 328983:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9522.680503] LustreError: 328983:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9526.119305] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9526.671839] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9531.571948] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 9539.029653] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9544.318700] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9551.462672] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9551.464187] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9551.476664] Lustre: Skipped 1 previous similar message [ 9553.721870] Lustre: server umount lustre-MDT0000 complete [ 9556.965768] LustreError: 328980:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9556.979112] LustreError: 328980:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9557.058570] LustreError: 328966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789644533 with bad export cookie 4469717441774573917 [ 9557.067678] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9557.488526] Lustre: server umount lustre-MDT0001 complete [ 9573.867364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9574.282917] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9574.344557] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9574.639030] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9578.769439] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 9587.788833] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9587.998829] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9588.048352] Lustre: 331572:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9588.555959] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9593.518351] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 9601.692523] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9607.935897] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9617.000918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9617.334050] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9617.350782] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9617.361522] LustreError: 333456:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 9617.383615] LustreError: 333456:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 9617.395042] LustreError: 333456:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9617.514589] Lustre: server umount lustre-OST0000 complete [ 9617.516882] LustreError: 333456:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9627.631620] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9627.928521] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9628.134979] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9636.033445] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 9638.225564] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9644.622193] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9653.914344] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9654.067836] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9654.155114] Lustre: lustre-OST0001: new disk, initializing [ 9654.160541] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9654.167882] Lustre: 334997:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9654.276793] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9662.601088] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9662.620961] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9662.768425] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9663.238766] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 9671.656313] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9678.310104] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9678.313764] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9678.328819] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9678.328827] Lustre: Skipped 1 previous similar message [ 9678.384997] Lustre: Skipped 1 previous similar message [ 9681.730573] Lustre: server umount lustre-OST0001 complete [ 9683.428431] LustreError: 333925:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9683.467950] LustreError: 333925:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9688.547688] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9688.552422] LustreError: 333925:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9688.556354] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9688.556362] Lustre: Skipped 1 previous similar message [ 9688.562554] Lustre: Skipped 1 previous similar message [ 9688.621268] LustreError: 333925:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9693.669772] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9693.670641] LustreError: 335320:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9693.686486] Lustre: Skipped 2 previous similar messages [ 9693.724289] LustreError: 335320:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9698.794875] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9698.806240] Lustre: Skipped 1 previous similar message [ 9700.319575] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9700.453542] Lustre: server umount lustre-OST0000 complete [ 9706.468956] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9706.475629] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9709.036361] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9709.039425] Lustre: Skipped 2 previous similar messages [ 9710.517843] Lustre: server umount lustre-MDT0000 complete [ 9714.147666] LustreError: 331586:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9714.179713] LustreError: 331586:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9714.721029] LustreError: 331567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789644691 with bad export cookie 4469717441774574470 [ 9714.729899] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9714.731950] LustreError: 331567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 9715.119969] Lustre: server umount lustre-MDT0001 complete [ 9725.446575] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 9728.516045] Key type lgssc unregistered [ 9728.969078] LNet: 336958:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9728.976551] LNetError: 336958:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9730.023235] LNet: Removed LNI 192.168.202.132@tcp [ 9730.985383] Key type .llcrypt unregistered [ 9730.987226] Key type ._llcrypt unregistered [ 9812.824347] Key type ._llcrypt registered [ 9812.826599] Key type .llcrypt registered [ 9812.983760] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [ 9828.326994] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 9829.808432] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9829.901423] alg: No test for adler32 (adler32-zlib) [ 9831.045404] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [ 9831.396797] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 9833.047919] Key type lgssc registered [ 9834.641483] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9872.655536] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9872.706314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9874.070694] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9874.100701] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9874.213284] Lustre: lustre-MDT0000: new disk, initializing [ 9874.300318] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9874.318774] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9879.466956] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 9890.668384] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9890.795722] Lustre: 341890:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9890.865847] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9890.869960] Lustre: Skipped 1 previous similar message [ 9890.999949] Lustre: lustre-MDT0001: new disk, initializing [ 9891.131131] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9891.196585] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9891.212026] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9895.604799] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 9905.564512] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9911.117677] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9917.714816] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9918.064602] Lustre: lustre-OST0000: new disk, initializing [ 9918.073453] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9918.084304] Lustre: 343858:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9918.236641] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9924.159195] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9924.166637] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9924.267240] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9924.921852] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 9935.993259] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9942.317890] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9942.738704] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9948.870471] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9949.082159] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9955.298466] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9955.301535] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9955.313106] LustreError: Skipped 1 previous similar message [ 9955.315648] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9955.341339] Lustre: Skipped 1 previous similar message [ 9957.730800] Lustre: server umount lustre-OST0000 complete [ 9962.980554] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9962.988773] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9963.011874] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9963.021941] Lustre: Skipped 1 previous similar message [ 9965.541932] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9965.545126] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9965.607527] Lustre: Skipped 1 previous similar message [ 9968.822855] Lustre: server umount lustre-MDT0000 complete [ 9972.864258] LustreError: 341883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789644949 with bad export cookie 629184004697348100 [ 9972.865757] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9972.870299] LustreError: 341883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9973.262157] Lustre: server umount lustre-MDT0001 complete [ 9982.346549] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 07:35:57 (1789644957) [ 9991.975919] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 07:36:06 (1789644966) [10000.468561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10000.895732] LustreError: 346616:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10000.916524] LustreError: 346616:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10000.975266] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10005.466506] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [10005.986033] LustreError: 346617:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10011.109403] LustreError: 346616:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10014.266351] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10014.746300] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10019.877889] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [10028.275112] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10035.533460] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10042.467828] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10042.782286] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10050.285381] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [10058.501850] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10065.973655] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10067.978728] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10069.517695] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10071.327881] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10073.207166] Lustre: Failing over lustre-MDT0000 [10073.520107] Lustre: server umount lustre-MDT0000 complete [10074.609916] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10074.621261] Lustre: Skipped 2 previous similar messages [10074.623389] LustreError: 346611:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10074.633737] LustreError: 346611:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10079.715766] LustreError: 348106:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10079.744205] LustreError: 348106:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10089.952349] Lustre: 340117:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789645050/real 1789645050] req@000000006bd31923 x1876578593218688/t0(0) o400->MGC192.168.202.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789645066 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10089.958148] LustreError: 346611:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10089.984376] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10090.011037] LustreError: 346611:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [10092.674860] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10100.616600] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10100.707047] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10102.185435] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10105.855354] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10105.903866] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [10105.943256] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10106.869902] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [10117.702988] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10119.648474] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10123.535519] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10125.458722] Lustre: Failing over lustre-MDT0000 [10125.957147] Lustre: server umount lustre-MDT0000 complete [10126.322719] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10126.361851] Lustre: Skipped 2 previous similar messages [10126.364203] LustreError: 346616:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10126.382297] LustreError: 346616:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 13 previous similar messages [10142.689612] Lustre: 340116:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789645102/real 1789645102] req@000000001a10eec6 x1876578593239296/t0(0) o400->MGC192.168.202.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789645118 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10142.740382] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10149.444212] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10153.582366] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10153.674989] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10158.493814] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10159.113966] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10159.139635] Lustre: Skipped 2 previous similar messages [10159.194520] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [10159.285768] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10160.200273] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [10174.087417] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10175.949967] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10184.683172] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10184.700879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10184.707728] Lustre: Skipped 2 previous similar messages [10184.729044] Lustre: Skipped 2 previous similar messages [10188.440222] Lustre: server umount lustre-MDT0000 complete [10189.813233] LustreError: 346617:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10189.847234] LustreError: 346617:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 26 previous similar messages [10194.743043] LustreError: 346598:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789645171 with bad export cookie 629184004697350809 [10194.776840] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10194.915463] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10195.371944] Lustre: server umount lustre-MDT0001 complete [10202.371959] Lustre: server umount lustre-OST0000 complete [10222.577521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10223.269156] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10228.912412] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [10238.749838] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10245.305541] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [10253.259358] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10259.195247] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10266.810689] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10268.218029] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10273.665184] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [10281.501055] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10293.730253] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10293.740240] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10293.761371] Lustre: Skipped 1 previous similar message [10293.765769] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10297.877419] Lustre: server umount lustre-OST0000 complete [10310.113378] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10310.119904] Lustre: Skipped 5 previous similar messages [10316.255230] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10316.450172] Lustre: server umount lustre-MDT0000 complete [10320.352905] LustreError: 352961:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10320.387368] LustreError: 352961:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [10320.491942] LustreError: 352940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789645296 with bad export cookie 629184004697351551 [10320.495202] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10320.516284] LustreError: 352940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10320.979469] Lustre: server umount lustre-MDT0001 complete [10333.043330] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [10336.101787] Key type lgssc unregistered [10336.456719] LNet: 356495:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10336.466687] LNetError: 356495:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10336.476933] LNet: Removed LNI 192.168.202.132@tcp [10337.367294] Key type .llcrypt unregistered [10337.373434] Key type ._llcrypt unregistered [10366.371811] Key type ._llcrypt registered [10366.379987] Key type .llcrypt registered [10366.572922] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [10367.987546] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10368.063180] alg: No test for adler32 (adler32-zlib) [10369.474183] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [10369.802809] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [10371.735343] Key type lgssc registered [10373.763768] Lustre: Echo OBD driver; http://www.lustre.org/ [10383.639424] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10383.657749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10385.183769] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10389.601143] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [10395.630379] LustreError: 358011:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10395.654834] LustreError: 358011:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10398.341450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10398.745056] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10403.516815] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [10411.796448] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10417.740200] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10424.230888] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10424.625992] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10429.763435] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10432.971796] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [10444.078945] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10458.597473] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10458.610733] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10458.623718] Lustre: Skipped 2 previous similar messages [10458.637285] Lustre: Skipped 2 previous similar messages [10463.717694] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10463.941741] Lustre: server umount lustre-MDT0000 complete [10467.807312] LustreError: 357991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789645444 with bad export cookie 147542650025774414 [10467.808836] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10467.810675] LustreError: 357991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10468.235804] Lustre: server umount lustre-MDT0001 complete [10482.118912] Lustre: server umount lustre-OST0000 complete [10497.853920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10498.320587] LustreError: 361743:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10498.440658] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10503.136658] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [10503.662515] LustreError: 361744:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10507.751503] LustreError: 361743:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10512.804869] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10512.871396] LustreError: 361744:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10513.213455] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10518.099478] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [10526.165578] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10532.953659] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10538.710582] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10539.169283] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10543.285801] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10546.220728] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [10554.284592] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10569.210832] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10569.236857] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10569.236866] Lustre: Skipped 2 previous similar messages [10569.251634] Lustre: Skipped 2 previous similar messages [10574.305053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10574.328621] Lustre: Skipped 4 previous similar messages [10574.879110] Lustre: server umount lustre-MDT0000 complete [10579.427601] LustreError: 361744:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10579.446383] LustreError: 361744:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10579.618064] LustreError: 361724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789645555 with bad export cookie 147542650025775527 [10579.629808] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10580.203652] Lustre: server umount lustre-MDT0001 complete [10595.376585] Lustre: server umount lustre-OST0000 complete [10612.829991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10613.598324] LustreError: 365655:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10613.767185] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10619.858585] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [10630.279530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10636.248480] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [10642.959431] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10648.634828] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10655.420935] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10655.740436] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10655.759301] Lustre: Skipped 1 previous similar message [10657.793421] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10662.066852] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [10669.174919] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10673.122879] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10673.125641] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10673.133940] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10673.150326] Lustre: Skipped 1 previous similar message [10678.660290] Lustre: server umount lustre-OST0000 complete [10682.336621] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10682.346379] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10682.362328] Lustre: Skipped 1 previous similar message [10682.367990] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10682.381922] Lustre: Skipped 2 previous similar messages [10688.531231] LustreError: 365650:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10688.562915] LustreError: 365650:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [10688.758583] Lustre: server umount lustre-MDT0000 complete [10692.593116] LustreError: 365635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789645668 with bad export cookie 147542650025776549 [10692.605246] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10692.615951] LustreError: 365635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [10693.204855] Lustre: server umount lustre-MDT0001 complete [10704.507488] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [10706.828818] Key type lgssc unregistered [10707.104816] LNet: 369187:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10707.109786] LNetError: 369187:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10708.135891] LNet: Removed LNI 192.168.202.132@tcp [10709.182983] Key type .llcrypt unregistered [10709.185267] Key type ._llcrypt unregistered [10733.277759] Key type ._llcrypt registered [10733.280677] Key type .llcrypt registered [10733.385294] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [10735.039739] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10735.064834] alg: No test for adler32 (adler32-zlib) [10736.072960] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [10736.256718] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [10737.919232] Key type lgssc registered [10738.984488] Lustre: Echo OBD driver; http://www.lustre.org/ [10746.501053] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10746.528927] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10746.544924] LustreError: 370624:0:(osd_handler.c:8546:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10746.569314] LustreError: 370624:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10746.574696] LustreError: 370624:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10746.579292] LustreError: 370624:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10746.584720] LustreError: 370624:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10753.843065] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10753.847697] LustreError: 371022:0:(osd_handler.c:8546:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10753.864756] LustreError: 371022:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [10753.869817] LustreError: 371022:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10753.875329] LustreError: 371022:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10753.884259] LustreError: 371022:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10773.745749] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [10776.937915] Key type lgssc unregistered [10777.224241] LNet: 372024:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10777.231471] LNetError: 372024:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10777.256796] LNet: Removed LNI 192.168.202.132@tcp [10778.169937] Key type .llcrypt unregistered [10778.174468] Key type ._llcrypt unregistered [10801.462995] Key type ._llcrypt registered [10801.465640] Key type .llcrypt registered [10801.641420] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [10822.189925] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [10823.175420] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10823.433639] alg: No test for adler32 (adler32-zlib) [10824.614392] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [10824.948321] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [10826.711700] Key type lgssc registered [10827.868171] Lustre: Echo OBD driver; http://www.lustre.org/ [10859.681943] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10859.731280] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10861.037974] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10861.099514] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10861.233787] Lustre: lustre-MDT0000: new disk, initializing [10861.281526] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10861.294528] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10866.305839] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [10877.491528] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10877.700600] Lustre: 375183:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10877.778978] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10877.782705] Lustre: Skipped 1 previous similar message [10877.883446] Lustre: lustre-MDT0001: new disk, initializing [10878.020635] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10878.082459] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10878.103625] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10882.401163] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [10891.910689] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10897.881947] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10904.396830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10904.635479] Lustre: lustre-OST0000: new disk, initializing [10904.642664] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10904.649513] Lustre: 377148:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10904.749898] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10910.423737] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [10910.819088] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10910.825346] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10910.965243] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10920.036380] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10923.786437] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10923.907876] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10927.988724] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10928.197881] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10931.684360] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10931.686703] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10931.696612] LustreError: Skipped 1 previous similar message [10931.701222] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10931.707528] Lustre: Skipped 1 previous similar message [10936.327344] Lustre: server umount lustre-OST0000 complete [10941.930908] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10941.937778] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10941.939699] Lustre: Skipped 1 previous similar message [10941.959104] Lustre: Skipped 2 previous similar messages [10948.310606] Lustre: server umount lustre-MDT0000 complete [10951.703082] LustreError: 375176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789645927 with bad export cookie 4281935560605637841 [10951.713247] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10951.715140] LustreError: 375176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10951.984383] Lustre: server umount lustre-MDT0001 complete [10960.310362] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [10962.264517] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 07:52:17 (1789645937) [10968.507802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10969.008608] LustreError: 379607:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10969.036137] LustreError: 379607:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10969.104848] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10973.375473] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [10974.178056] LustreError: 379608:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10979.298693] LustreError: 379607:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10981.024445] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10981.331954] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10985.210909] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [10991.842426] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10997.211434] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11005.059110] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11005.465983] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11012.626031] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [11020.601251] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11027.711721] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11027.914209] Lustre: lustre-OST0001: new disk, initializing [11027.917587] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11027.923719] Lustre: 382441:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11028.000177] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11034.228243] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [11034.257687] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11034.270697] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11034.373812] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11043.842622] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11049.952071] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11049.959048] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11049.971502] LustreError: Skipped 1 previous similar message [11049.974427] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11050.021806] Lustre: Skipped 1 previous similar message [11053.657947] Lustre: server umount lustre-OST0001 complete [11055.082570] LustreError: 381430:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11055.123536] LustreError: 381430:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11060.196335] LustreError: 381430:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11060.240359] LustreError: 381430:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11070.437801] LustreError: 381429:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11070.485763] LustreError: 381429:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11072.992571] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11073.007336] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11073.026179] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11073.031065] Lustre: Skipped 2 previous similar messages [11078.734826] Lustre: server umount lustre-OST0000 complete [11083.747794] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11083.755353] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11083.788459] Lustre: Skipped 1 previous similar message [11083.799163] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11083.806819] Lustre: Skipped 2 previous similar messages [11088.899689] Lustre: server umount lustre-MDT0000 complete [11090.912373] LustreError: 379608:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11090.936158] LustreError: 379608:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11092.678992] LustreError: 379587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789646068 with bad export cookie 4281935560605638723 [11092.705373] LustreError: 379587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11092.716548] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11099.118981] Lustre: server umount lustre-MDT0001 complete [11108.932357] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [11111.651195] Key type lgssc unregistered [11112.010351] LNet: 384427:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11112.024821] LNetError: 384427:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11112.042539] LNet: Removed LNI 192.168.202.132@tcp [11112.861617] Key type .llcrypt unregistered [11112.863770] Key type ._llcrypt unregistered [11162.699274] Key type ._llcrypt registered [11162.705594] Key type .llcrypt registered [11162.810931] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [11164.346305] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11164.378852] alg: No test for adler32 (adler32-zlib) [11165.503612] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [11165.825195] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [11167.559333] Key type lgssc registered [11168.962666] Lustre: Echo OBD driver; http://www.lustre.org/ [11177.324628] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11177.366883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11178.776324] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11178.814893] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11179.079118] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11184.214284] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [11193.789352] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11193.918655] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11193.956487] Lustre: 387226:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11194.204342] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11199.039703] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [11206.583648] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11212.161270] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11214.817846] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11214.826631] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11214.849714] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11217.897845] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11217.903424] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11217.921318] Lustre: Skipped 1 previous similar message [11220.503526] Lustre: server umount lustre-MDT0000 complete [11224.236777] LustreError: 387220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789646200 with bad export cookie 16185220868654229401 [11224.240097] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11224.257095] LustreError: 387220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11224.700552] Lustre: server umount lustre-MDT0001 complete [11234.820715] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 07:56:49 (1789646209) [11241.311456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11241.845019] LustreError: 389737:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11241.862739] LustreError: 389737:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11241.923731] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11246.259994] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [11247.071947] LustreError: 389738:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11252.197082] LustreError: 389737:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11253.689701] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11254.048841] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11257.950426] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [11266.783876] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11272.312347] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11278.690250] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11278.842685] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11279.012365] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11285.040130] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [11292.479350] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11301.675069] Lustre: Permanently deactivating lustre-OST0000 [11301.684749] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11307.587399] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11310.115628] LustreError: 392674:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11310.139748] LustreError: 392674:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11310.157772] LustreError: 392674:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11313.942431] LustreError: 392722:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11313.951924] LustreError: 392722:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11322.539549] Lustre: server umount lustre-OST0000 complete [11325.828933] LustreError: 392972:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11325.836323] LustreError: 392972:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11331.040809] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11331.051953] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11331.062437] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11336.678472] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11336.685756] Lustre: Skipped 2 previous similar messages [11338.842100] Lustre: server umount lustre-MDT0000 complete [11341.793767] LustreError: 389732:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11341.842866] LustreError: 389732:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11343.132304] LustreError: 392426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789646319 with bad export cookie 16185220868654229982 [11343.137120] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11343.140364] LustreError: 392426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [11353.787387] Lustre: server umount lustre-MDT0001 complete [11364.811647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11371.040779] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [11374.300339] LustreError: 394131:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11382.162596] Lustre: 394470:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.132@tcp [11384.963254] Lustre: server umount MGS complete [11397.980133] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11398.633677] LustreError: 395194:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11398.663494] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11398.674121] Lustre: Skipped 1 previous similar message [11398.729293] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11403.113305] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [11408.887631] LustreError: 395195:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11408.926536] LustreError: 395195:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11416.201127] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11416.627697] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11416.823098] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11422.433244] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [11430.547932] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11436.794680] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11438.715468] Lustre: Permanently reactivating lustre-OST0000 [11438.726748] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11438.732259] Lustre: Skipped 2 previous similar messages [11467.233389] LustreError: 396975:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11475.280513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11475.809437] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11483.122448] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [11492.665180] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11502.561778] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11502.563824] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11502.576312] LustreError: Skipped 1 previous similar message [11502.578799] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11502.600050] Lustre: Skipped 2 previous similar messages [11502.617259] Lustre: Skipped 1 previous similar message [11505.853561] Lustre: server umount lustre-OST0000 complete [11512.803732] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11512.812050] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11512.815045] Lustre: Skipped 1 previous similar message [11515.901372] Lustre: server umount lustre-MDT0000 complete [11517.920094] LustreError: 395924:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11517.945897] LustreError: 395924:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11519.790251] LustreError: 395176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789646496 with bad export cookie 16185220868654231032 [11519.810478] LustreError: 395176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11519.816912] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11520.115716] Lustre: server umount lustre-MDT0001 complete [11532.076293] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [11535.077969] Key type lgssc unregistered [11535.548066] LNet: 399572:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11535.562858] LNetError: 399572:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11535.590156] LNet: Removed LNI 192.168.202.132@tcp [11536.559373] Key type .llcrypt unregistered [11536.562080] Key type ._llcrypt unregistered [11557.400928] Key type ._llcrypt registered [11557.410862] Key type .llcrypt registered [11557.535474] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [11572.314753] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [11573.097773] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11573.189329] alg: No test for adler32 (adler32-zlib) [11574.234970] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [11574.445324] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [11576.095237] Key type lgssc registered [11577.226216] Lustre: Echo OBD driver; http://www.lustre.org/ [11609.603652] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11609.631264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11610.903538] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11610.939342] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11611.003587] Lustre: lustre-MDT0000: new disk, initializing [11611.078294] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11611.100774] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11615.022483] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [11624.753863] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11624.858298] Lustre: 402728:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11624.901378] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11624.905250] Lustre: Skipped 1 previous similar message [11624.966703] Lustre: lustre-MDT0001: new disk, initializing [11625.042723] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11625.091119] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11625.109980] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11630.839514] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [11640.336129] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11646.720370] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11653.969224] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11654.412339] Lustre: lustre-OST0000: new disk, initializing [11654.415962] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11654.421081] Lustre: 404695:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11654.502057] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11660.743981] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [11662.447897] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11662.466058] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11662.588699] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11670.379604] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11675.331904] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11675.485217] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11680.234694] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11680.474991] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11683.301258] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11683.306610] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11683.317535] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11683.357599] Lustre: Skipped 1 previous similar message [11688.422085] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11688.432610] Lustre: Skipped 2 previous similar messages [11688.757293] Lustre: server umount lustre-OST0000 complete [11693.537122] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11693.548115] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11693.572507] Lustre: Skipped 1 previous similar message [11693.599288] Lustre: Skipped 1 previous similar message [11698.659746] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11698.677024] Lustre: Skipped 1 previous similar message [11698.948497] Lustre: server umount lustre-MDT0000 complete [11703.163215] LustreError: 404693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789646679 with bad export cookie 2397435603458732369 [11703.171102] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11703.172350] LustreError: 404693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11703.739589] Lustre: server umount lustre-MDT0001 complete [11713.600877] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11715.228324] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 08:04:50 (1789646690) [11723.034670] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11723.680066] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11728.865726] LustreError: 407156:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11728.887467] LustreError: 407156:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11728.891933] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [11732.971618] LustreError: 407156:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11738.086559] LustreError: 407155:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11740.240458] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11740.723651] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11745.846633] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [11754.017288] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11759.696527] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11766.161536] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11766.646362] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11772.931534] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [11780.326936] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11782.436337] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [11785.044499] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [11794.413867] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11794.421950] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11794.437559] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11795.423982] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11800.495260] Lustre: server umount lustre-OST0000 complete [11805.664634] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11805.685519] Lustre: Skipped 1 previous similar message [11805.697149] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11805.705623] Lustre: Skipped 3 previous similar messages [11810.330773] Lustre: server umount lustre-MDT0000 complete [11810.787261] LustreError: 407152:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11814.171244] LustreError: 407137:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789646790 with bad export cookie 2397435603458733251 [11814.181417] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11814.186304] LustreError: 407137:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11814.669576] Lustre: server umount lustre-MDT0001 complete [11826.193963] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [11829.527636] Key type lgssc unregistered [11829.966884] LNet: 410882:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11829.973481] LNetError: 410882:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11831.017279] LNet: Removed LNI 192.168.202.132@tcp [11832.183682] Key type .llcrypt unregistered [11832.185574] Key type ._llcrypt unregistered [11859.422835] Key type ._llcrypt registered [11859.429669] Key type .llcrypt registered [11859.565834] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [11859.864426] Key type .llcrypt unregistered [11859.869440] Key type ._llcrypt unregistered [11877.349646] Key type ._llcrypt registered [11877.352495] Key type .llcrypt registered [11877.517984] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [11878.648523] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11878.906481] alg: No test for adler32 (adler32-zlib) [11879.905576] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [11880.205860] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [11881.911276] Key type lgssc registered [11883.056698] Lustre: Echo OBD driver; http://www.lustre.org/ [11892.227892] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11892.259264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11893.663750] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11898.839745] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [11902.946387] LustreError: 413090:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11902.959557] LustreError: 413090:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11908.075784] LustreError: 413091:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11909.432252] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11909.971356] LustreError: 413090:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11910.026712] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11915.235303] LustreError: 413087:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11915.253894] LustreError: 413087:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11917.445708] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [11918.343616] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11923.428457] LustreError: 413085:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11923.452144] LustreError: 413085:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11925.340435] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11933.235317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11933.765779] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11939.451611] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [11949.031133] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11949.033876] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11949.038405] LustreError: Skipped 1 previous similar message [11949.039166] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11949.065486] Lustre: Skipped 1 previous similar message [11954.149827] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11954.156169] Lustre: Skipped 1 previous similar message [11954.850505] Lustre: server umount lustre-OST0000 complete [11959.265935] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11959.266504] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11959.279126] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11959.301494] Lustre: Skipped 1 previous similar message [11964.388134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11964.399929] Lustre: Skipped 1 previous similar message [11965.013882] Lustre: server umount lustre-MDT0000 complete [11968.762478] LustreError: 413071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789646945 with bad export cookie 15314614450676099834 [11968.764568] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11968.774083] LustreError: 413071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11969.158633] Lustre: server umount lustre-MDT0001 complete [11978.765263] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [11980.936731] Key type lgssc unregistered [11981.120480] LNet: 416238:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11981.129991] LNetError: 416238:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11981.142919] LNet: Removed LNI 192.168.202.132@tcp [11981.866285] Key type .llcrypt unregistered [11981.871238] Key type ._llcrypt unregistered [12007.024384] Key type ._llcrypt registered [12007.027927] Key type .llcrypt registered [12007.153771] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [12007.981281] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12008.003839] alg: No test for adler32 (adler32-zlib) [12009.222349] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [12009.491872] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [12011.263218] Key type lgssc registered [12012.351474] Lustre: Echo OBD driver; http://www.lustre.org/ [12020.357783] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12020.392250] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12035.999525] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12041.119568] LustreError: 417624:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.132@tcp: failed processing log, type 4: rc = -110 [12067.871220] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12075.303613] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [12082.942480] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12089.835339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12090.466673] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12090.511919] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12095.467782] LustreError: 417649:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12095.491132] LustreError: 417649:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12095.509326] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [12100.581479] LustreError: 417650:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12102.635621] LustreError: 417649:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12102.653032] LustreError: 417649:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12104.705374] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12105.190230] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12110.646890] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [12118.619312] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12125.294881] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12135.906241] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12135.909884] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12135.948650] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12135.958340] Lustre: Skipped 1 previous similar message [12140.410418] Lustre: server umount lustre-OST0000 complete [12146.161260] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12146.163428] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12146.181984] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12146.220575] Lustre: Skipped 2 previous similar messages [12150.709095] Lustre: server umount lustre-MDT0000 complete [12151.266041] LustreError: 418667:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12151.291698] LustreError: 418667:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12155.449342] LustreError: 417631:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789647131 with bad export cookie 15730142099388110592 [12155.459692] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12155.770812] Lustre: server umount lustre-MDT0001 complete [12168.419866] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [12171.374412] Key type lgssc unregistered [12171.733548] LNet: 421195:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12171.743381] LNetError: 421195:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12171.765751] LNet: Removed LNI 192.168.202.132@tcp [12172.803718] Key type .llcrypt unregistered [12172.805501] Key type ._llcrypt unregistered [12198.970543] Key type ._llcrypt registered [12198.972601] Key type .llcrypt registered [12199.091205] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [12200.272107] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12200.361664] alg: No test for adler32 (adler32-zlib) [12201.472915] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [12201.679563] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [12203.383683] Key type lgssc registered [12204.594991] Lustre: Echo OBD driver; http://www.lustre.org/ [12213.365616] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12213.389804] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12215.317603] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12221.348187] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [12225.510153] LustreError: 422613:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12225.525552] LustreError: 422613:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12230.627070] LustreError: 422614:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12231.186357] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12231.770715] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12236.530570] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [12244.560256] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12251.717601] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12260.242034] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12260.979549] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12269.441771] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [12277.790811] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12285.571286] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12293.090101] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12293.106203] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12293.126081] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12296.176471] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12296.179410] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12296.188757] Lustre: Skipped 1 previous similar message [12296.198703] Lustre: Skipped 3 previous similar messages [12297.844595] Lustre: server umount lustre-MDT0000 complete [12298.653375] LustreError: 422608:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12300.386563] LustreError: 422594:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.32@tcp arrived at 1789647276 with bad export cookie 5849041164737013217 [12300.409366] LustreError: 422594:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12301.297881] LustreError: 422613:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12301.323599] LustreError: 422613:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12302.326272] LustreError: 422595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789647278 with bad export cookie 5849041164737012552 [12302.342677] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12306.400413] LustreError: 422614:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12306.409164] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12306.431416] LustreError: 422614:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12306.460217] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12308.552366] Lustre: server umount lustre-OST0000 complete [12315.330753] Lustre: server umount lustre-MDT0001 complete [12326.482668] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [12329.237393] Key type lgssc unregistered [12329.499516] LNet: 426319:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12329.504317] LNetError: 426319:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12329.524071] LNet: Removed LNI 192.168.202.132@tcp [12330.600967] Key type .llcrypt unregistered [12330.603584] Key type ._llcrypt unregistered [12355.830964] Key type ._llcrypt registered [12355.833278] Key type .llcrypt registered [12355.999621] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [12357.246040] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12357.281854] alg: No test for adler32 (adler32-zlib) [12358.488311] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [12358.828256] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [12360.555146] Key type lgssc registered [12362.053730] Lustre: Echo OBD driver; http://www.lustre.org/ [12372.394105] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12372.405289] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12373.942391] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12379.365436] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [12384.232857] LustreError: 427729:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12384.251859] LustreError: 427729:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12389.345186] LustreError: 427730:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12390.339372] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12390.624842] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12395.580885] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [12404.491506] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12409.815859] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12416.445482] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12416.790087] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12418.876784] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12423.200731] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [12430.108918] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12436.960727] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12436.967197] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12436.979649] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12437.916532] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.32@tcp (stopping) [12439.534320] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12439.542038] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12439.581972] Lustre: Skipped 1 previous similar message [12443.035436] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.32@tcp (stopping) [12448.158481] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.32@tcp (stopping) [12448.162914] Lustre: Skipped 2 previous similar messages [12449.251918] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12449.447724] Lustre: server umount lustre-MDT0001 complete [12449.764290] LustreError: 427725:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12449.790501] LustreError: 427725:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12454.882359] LustreError: 427724:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12454.904375] LustreError: 427724:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12460.005917] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12460.009688] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12460.017292] LustreError: 427730:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12460.026602] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12462.352713] Lustre: server umount lustre-OST0000 complete [12467.181450] Lustre: server umount lustre-MDT0000 complete [12481.432954] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [12483.961094] Key type lgssc unregistered [12484.194738] LNet: 431414:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12484.207430] LNetError: 431414:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12484.235094] LNet: Removed LNI 192.168.202.132@tcp [12485.244142] Key type .llcrypt unregistered [12485.247153] Key type ._llcrypt unregistered [12510.176167] Key type ._llcrypt registered [12510.178267] Key type .llcrypt registered [12510.298165] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [12528.088821] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [12529.498940] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12529.534578] alg: No test for adler32 (adler32-zlib) [12530.623638] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [12530.846741] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [12532.528889] Key type lgssc registered [12533.746467] Lustre: Echo OBD driver; http://www.lustre.org/ [12576.605303] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [12592.568376] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12592.598359] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12594.007467] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12594.041221] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12594.163525] Lustre: lustre-MDT0000: new disk, initializing [12594.307382] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12594.331335] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12600.475339] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [12619.418853] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12619.737443] Lustre: lustre-OST0000: new disk, initializing [12619.746804] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12619.753637] Lustre: Skipped 1 previous similar message [12619.762581] Lustre: 436448:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12619.853995] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12626.037649] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12626.048440] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12626.116079] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12626.739973] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [12638.648816] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12656.946813] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12657.087056] Lustre: 435431:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12657.181794] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12657.247950] Lustre: lustre-MDT0001: new disk, initializing [12657.367435] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12657.398933] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [12657.406418] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [12661.466840] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [12672.777474] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [12673.153809] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12680.094492] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [12680.490835] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [12687.826166] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.32@tcp (stopping) [12688.353586] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12688.391562] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12688.403982] Lustre: Skipped 1 previous similar message [12692.915959] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.32@tcp (stopping) [12693.721869] Lustre: server umount lustre-MDT0001 complete [12703.714853] LustreError: 435441:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12703.726745] LustreError: 435441:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [12708.841338] LustreError: 435438:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12708.868794] LustreError: 435438:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12712.940814] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12712.954647] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12712.971536] Lustre: Skipped 1 previous similar message [12712.974117] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12712.978077] Lustre: Skipped 2 previous similar messages [12713.963335] LustreError: 435441:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12718.313658] Lustre: server umount lustre-OST0000 complete [12719.076753] LustreError: 437777:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12728.911167] Lustre: server umount lustre-MDT0000 complete [12744.465196] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [12747.929110] Key type lgssc unregistered [12748.315608] LNet: 440203:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12748.321172] LNetError: 440203:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12748.343593] LNet: Removed LNI 192.168.202.132@tcp [12749.522579] Key type .llcrypt unregistered [12749.524649] Key type ._llcrypt unregistered [12795.029104] Key type ._llcrypt registered [12795.030701] Key type .llcrypt registered [12795.130651] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [12795.523566] Key type .llcrypt unregistered [12795.528707] Key type ._llcrypt unregistered [12815.499937] Key type ._llcrypt registered [12815.509566] Key type .llcrypt registered [12815.740604] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [12817.339555] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12817.490835] alg: No test for adler32 (adler32-zlib) [12818.601779] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [12818.942559] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [12820.778649] Key type lgssc registered [12822.437244] Lustre: Echo OBD driver; http://www.lustre.org/ [12850.243997] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12850.260627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12851.505380] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12851.531559] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12851.610516] Lustre: lustre-MDT0000: new disk, initializing [12851.699084] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12851.713339] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12856.122515] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [12866.962823] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12867.089822] Lustre: 443572:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12867.128468] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12867.131755] Lustre: Skipped 1 previous similar message [12867.190366] Lustre: lustre-MDT0001: new disk, initializing [12867.272331] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12867.312556] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12867.323634] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12872.476623] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [12884.331214] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12891.683649] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12898.495721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12898.913043] Lustre: lustre-OST0000: new disk, initializing [12898.920474] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12898.930432] Lustre: 445541:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12899.038603] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12905.277831] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [12905.567836] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12905.575748] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12905.712448] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12914.918446] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12923.872747] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12923.886173] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12923.897481] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12926.439304] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12926.449765] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12926.467875] Lustre: Skipped 1 previous similar message [12927.775356] Lustre: server umount lustre-MDT0000 complete [12931.364504] LustreError: 444368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789647907 with bad export cookie 2580168019302548244 [12931.371349] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12931.378070] LustreError: 444368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12931.563755] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12931.581549] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12931.591293] Lustre: Skipped 2 previous similar messages [12931.802741] Lustre: server umount lustre-MDT0001 complete [12937.697818] Lustre: server umount lustre-OST0000 complete [12947.572954] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 08:25:22 (1789647922) [12956.307350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12957.169482] LustreError: 447568:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12957.198666] LustreError: 447568:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12957.302557] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12962.272100] LustreError: 447569:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12963.255585] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [12967.414281] LustreError: 447568:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12972.523072] LustreError: 447569:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12975.820790] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12976.515020] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12983.704303] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [12990.099646] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12997.064862] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13018.411314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13018.674511] Lustre: Found index 0 for lustre-OST0000, updating log [13018.873523] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13026.218768] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [13029.111063] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13035.782939] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13044.705133] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13044.712526] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13044.729733] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13048.738692] Lustre: server umount lustre-OST0000 complete [13064.630271] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13064.648636] Lustre: Skipped 1 previous similar message [13064.652485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13064.657056] Lustre: Skipped 4 previous similar messages [13070.841899] Lustre: server umount lustre-MDT0000 complete [13071.846829] LustreError: 447564:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13071.866714] LustreError: 447564:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13074.388910] LustreError: 447548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789648050 with bad export cookie 2580168019302549637 [13074.400341] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13074.403620] LustreError: 447548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13074.664530] Lustre: server umount lustre-MDT0001 complete [13082.167426] LNet: 1 local NIs in recovery (showing 1): 192.168.202.132@tcp [13084.960350] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 08:27:40 (1789648060) [13711.368946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13712.054471] LustreError: 453982:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13712.073489] LustreError: 453982:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13712.192954] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13717.472231] LustreError: 453983:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13718.393697] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [13722.602415] LustreError: 453982:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13728.756401] LustreError: 453983:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13728.784721] LustreError: 453983:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13728.985698] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13729.494932] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13734.663995] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [13743.670043] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13750.138938] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13771.129837] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13771.426899] Lustre: Found index 0 for lustre-OST0000, updating log [13771.579739] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13773.409805] LNet: 1 local NIs in recovery (showing 1): 192.168.202.132@tcp [13777.848835] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [13788.024605] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13791.203717] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13791.220719] Lustre: Skipped 1 previous similar message [13791.252156] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13791.270092] Lustre: Skipped 2 previous similar messages [13796.564823] Lustre: server umount lustre-OST0000 complete [13802.610788] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13802.625110] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13802.629937] Lustre: Skipped 1 previous similar message [13802.658619] Lustre: Skipped 1 previous similar message [13804.710041] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13804.715686] Lustre: Skipped 3 previous similar messages [13807.950321] Lustre: server umount lustre-MDT0000 complete [13808.805446] LustreError: 453983:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13808.820840] LustreError: 453983:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13812.545672] LustreError: 455835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789648788 with bad export cookie 2580168019302550722 [13812.552410] LustreError: 455835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13812.553123] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13812.985670] Lustre: server umount lustre-MDT0001 complete [13832.800672] LNet: 1 local NIs in recovery (showing 1): 192.168.202.132@tcp [13846.649267] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [13858.647731] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [13898.275701] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 08:41:13 (1789648873) [13917.220806] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [13920.467981] Key type lgssc unregistered [13920.819559] LNet: 460604:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13920.849940] LNetError: 460604:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14034.545544] LNet: Removed LNI 192.168.202.132@tcp [14036.020459] Key type .llcrypt unregistered [14036.027112] Key type ._llcrypt unregistered [14053.907500] Key type ._llcrypt registered [14053.909427] Key type .llcrypt registered [14053.990678] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [14055.675687] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14055.773094] alg: No test for adler32 (adler32-zlib) [14056.759902] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [14057.026950] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [14058.696513] Key type lgssc registered [14060.009916] Lustre: Echo OBD driver; http://www.lustre.org/ [14065.431365] LNet: Added LNI 192.168.202.132@tcp100 [8/256/0/180] [14066.547101] LNet: Added LNI 192.168.202.132@tcp101 [8/256/0/180] [14067.785512] LNet: Added LNI 192.168.202.132@tcp102 [8/256/0/180] [14068.897827] LNet: Added LNI 192.168.202.132@tcp103 [8/256/0/180] [14069.814481] LNet: Added LNI 192.168.202.132@tcp104 [8/256/0/180] [14070.764107] LNet: Added LNI 192.168.202.132@tcp105 [8/256/0/180] [14071.584749] LNet: Added LNI 192.168.202.132@tcp106 [8/256/0/180] [14072.533540] LNet: Added LNI 192.168.202.132@tcp107 [8/256/0/180] [14073.615882] LNet: Added LNI 192.168.202.132@tcp108 [8/256/0/180] [14074.553822] LNet: Added LNI 192.168.202.132@tcp109 [8/256/0/180] [14075.546436] LNet: Added LNI 192.168.202.132@tcp110 [8/256/0/180] [14076.528516] LNet: Added LNI 192.168.202.132@tcp111 [8/256/0/180] [14077.451926] LNet: Added LNI 192.168.202.132@tcp112 [8/256/0/180] [14078.481310] LNet: Added LNI 192.168.202.132@tcp113 [8/256/0/180] [14079.604255] LNet: Added LNI 192.168.202.132@tcp114 [8/256/0/180] [14080.468673] LNet: Added LNI 192.168.202.132@tcp115 [8/256/0/180] [14081.432995] LNet: Added LNI 192.168.202.132@tcp116 [8/256/0/180] [14082.354759] LNet: Added LNI 192.168.202.132@tcp117 [8/256/0/180] [14083.270709] LNet: Added LNI 192.168.202.132@tcp118 [8/256/0/180] [14084.330958] LNet: Added LNI 192.168.202.132@tcp119 [8/256/0/180] [14085.367889] LNet: Added LNI 192.168.202.132@tcp120 [8/256/0/180] [14086.278871] LNet: Added LNI 192.168.202.132@tcp121 [8/256/0/180] [14087.271328] LNet: Added LNI 192.168.202.132@tcp122 [8/256/0/180] [14088.114859] LNet: Added LNI 192.168.202.132@tcp123 [8/256/0/180] [14089.102949] LNet: Added LNI 192.168.202.132@tcp124 [8/256/0/180] [14090.274366] LNet: Added LNI 192.168.202.132@tcp125 [8/256/0/180] [14091.036096] LNet: Added LNI 192.168.202.132@tcp126 [8/256/0/180] [14092.003972] LNet: Added LNI 192.168.202.132@tcp127 [8/256/0/180] [14093.068221] LNet: Added LNI 192.168.202.132@tcp128 [8/256/0/180] [14093.969226] LNet: Added LNI 192.168.202.132@tcp129 [8/256/0/180] [14094.925611] LNet: Added LNI 192.168.202.132@tcp130 [8/256/0/180] [14095.954667] LNet: Added LNI 192.168.202.132@tcp131 [8/256/0/180] [14096.824780] LNet: Added LNI 192.168.202.132@tcp132 [8/256/0/180] [14097.719652] LNet: Added LNI 192.168.202.132@tcp133 [8/256/0/180] [14098.870517] LNet: Added LNI 192.168.202.132@tcp134 [8/256/0/180] [14099.617290] LNet: Added LNI 192.168.202.132@tcp135 [8/256/0/180] [14105.980357] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14106.000260] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14107.158527] Lustre: 463631:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.132@tcp: can't add NID 192.168.202.132@tcp130: rc = -75 [14107.230392] LNetError: 461091:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.132 [14107.247098] LNetError: Protocol error connecting to 192.168.202.132@tcp112 at host 192.168.202.132:988: is it running a compatible version of Lustre? [14107.255893] LNetError: Skipped 1 previous similar message [14107.444477] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14107.483355] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14107.618392] Lustre: lustre-MDT0000: new disk, initializing [14107.711814] Lustre: 463687:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.132@tcp: can't add NID 192.168.202.132@tcp130: rc = -75 [14107.727892] Lustre: 463687:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14107.759612] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14107.786600] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14108.255433] LNet: 6 local NIs in recovery (showing 6): 192.168.202.132@tcp125, 192.168.202.132@tcp126, 192.168.202.132@tcp128, 192.168.202.132@tcp129, 192.168.202.132@tcp127, 192.168.202.132@tcp124 [14108.257317] LNetError: 463652:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.132 [14108.290716] LNetError: 463652:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 10 previous similar messages [14108.298745] LNetError: Protocol error connecting to 192.168.202.132@tcp107 at host 192.168.202.132:988: is it running a compatible version of Lustre? [14108.313101] LNetError: Skipped 2 previous similar messages [14111.729984] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [14122.967157] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14123.172901] Lustre: 463647:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14123.228550] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14123.233805] Lustre: Skipped 1 previous similar message [14123.251559] Lustre: 464451:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.132@tcp: can't add NID 192.168.202.132@tcp130: rc = -75 [14123.254918] Lustre: 464451:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14123.265627] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [14123.316304] Lustre: lustre-MDT0001: new disk, initializing [14123.539234] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14123.610339] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14123.633882] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14128.613023] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [14137.955523] LNetError: 461091:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.132 [14137.976712] LNetError: Protocol error connecting to 192.168.202.132@tcp123 at host 192.168.202.132:988: is it running a compatible version of Lustre? [14139.792729] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14140.006183] LNetError: 461092:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.132 [14140.021966] LNetError: Protocol error connecting to 192.168.202.132@tcp121 at host 192.168.202.132:988: is it running a compatible version of Lustre? [14145.120266] LNetError: 461090:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.132 [14145.141478] LNetError: Protocol error connecting to 192.168.202.132@tcp122 at host 192.168.202.132:988: is it running a compatible version of Lustre? [14147.102479] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14153.318409] LNetError: 463658:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.132 [14153.332560] LNetError: 463658:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 4 previous similar messages [14153.359075] LNetError: Protocol error connecting to 192.168.202.132@tcp121 at host 192.168.202.132:988: is it running a compatible version of Lustre? [14153.380453] LNetError: Skipped 4 previous similar messages [14157.154299] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14157.649978] Lustre: lustre-OST0000: new disk, initializing [14157.653700] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14157.673290] Lustre: 465625:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14157.799890] Lustre: 465630:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.132@tcp: can't add NID 192.168.202.132@tcp130: rc = -75 [14157.812263] Lustre: 465630:0:(lustre_peer.c:124:class_add_uuid()) Skipped 17 previous similar messages [14157.880392] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14166.401707] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [14166.586333] Lustre: 465984:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.132@tcp: can't add NID 192.168.202.132@tcp130: rc = -75 [14166.620569] Lustre: 465984:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14166.727184] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14166.739785] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14166.987291] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14170.720987] LNetError: 463654:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.132 [14170.741489] LNetError: 463654:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 6 previous similar messages [14170.761382] LNetError: Protocol error connecting to 192.168.202.132@tcp122 at host 192.168.202.132:988: is it running a compatible version of Lustre? [14170.788902] LNetError: Skipped 6 previous similar messages [14180.243799] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14192.619882] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14192.621253] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14192.628693] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14192.640330] LustreError: Skipped 1 previous similar message [14197.735064] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14197.746884] Lustre: Skipped 2 previous similar messages [14202.859219] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14202.870090] Lustre: Skipped 1 previous similar message [14203.359294] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14203.620766] Lustre: server umount lustre-OST0000 complete [14204.580064] LNetError: 461089:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.132 [14204.588292] LNetError: 461089:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 10 previous similar messages [14204.597228] LNetError: Protocol error connecting to 192.168.202.132@tcp121 at host 192.168.202.132:988: is it running a compatible version of Lustre? [14204.611198] LNetError: Skipped 10 previous similar messages [14207.970474] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14207.996858] Lustre: Skipped 1 previous similar message [14208.004689] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14208.015576] Lustre: Skipped 1 previous similar message [14218.211199] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14218.214701] Lustre: Skipped 3 previous similar messages [14221.279401] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14221.691812] Lustre: server umount lustre-MDT0000 complete [14226.533912] LustreError: 463641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789649202 with bad export cookie 9622983604185761981 [14226.536942] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14226.546431] LustreError: 463641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14227.023268] Lustre: server umount lustre-MDT0001 complete [14239.764524] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [14242.615182] Key type lgssc unregistered [14243.131312] LNet: 467542:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14243.162973] LNetError: 467542:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14244.192377] LNet: Removed LNI 192.168.202.132@tcp [14244.206509] LNetError: 467542:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14244.213835] LNet: Removed LNI 192.168.202.132@tcp100 [14245.279177] LNet: Removed LNI 192.168.202.132@tcp101 [14245.280798] LNetError: 467542:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14245.284744] LNetError: 467542:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14245.294572] LNet: Removed LNI 192.168.202.132@tcp102 [14246.303871] LNet: Removed LNI 192.168.202.132@tcp103 [14247.327259] LNet: Removed LNI 192.168.202.132@tcp104 [14247.343768] LNetError: 467542:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14247.361848] LNetError: 467542:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14247.375066] LNet: Removed LNI 192.168.202.132@tcp105 [14248.415300] LNet: Removed LNI 192.168.202.132@tcp106 [14249.439191] LNet: Removed LNI 192.168.202.132@tcp107 [14250.463372] LNet: Removed LNI 192.168.202.132@tcp108 [14251.488882] LNet: Removed LNI 192.168.202.132@tcp109 [14251.498566] LNetError: 467542:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14251.514020] LNetError: 467542:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [14251.532213] LNet: Removed LNI 192.168.202.132@tcp110 [14252.575171] LNet: Removed LNI 192.168.202.132@tcp111 [14253.599164] LNet: Removed LNI 192.168.202.132@tcp112 [14254.623202] LNet: Removed LNI 192.168.202.132@tcp113 [14255.711220] LNet: Removed LNI 192.168.202.132@tcp114 [14256.738544] LNet: Removed LNI 192.168.202.132@tcp115 [14257.759180] LNet: Removed LNI 192.168.202.132@tcp116 [14258.786904] LNet: Removed LNI 192.168.202.132@tcp117 [14259.810062] LNet: Removed LNI 192.168.202.132@tcp118 [14259.812162] LNetError: 467542:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14259.835350] LNetError: 467542:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [14259.851058] LNet: Removed LNI 192.168.202.132@tcp119 [14260.895343] LNet: Removed LNI 192.168.202.132@tcp120 [14261.919263] LNet: Removed LNI 192.168.202.132@tcp121 [14262.943982] LNet: Removed LNI 192.168.202.132@tcp122 [14263.968767] LNet: Removed LNI 192.168.202.132@tcp123 [14264.991846] LNet: Removed LNI 192.168.202.132@tcp124 [14266.015223] LNet: Removed LNI 192.168.202.132@tcp125 [14267.039193] LNet: Removed LNI 192.168.202.132@tcp126 [14268.063235] LNet: Removed LNI 192.168.202.132@tcp127 [14269.087262] LNet: Removed LNI 192.168.202.132@tcp128 [14270.111149] LNet: Removed LNI 192.168.202.132@tcp129 [14270.118409] LNet: Removed LNI 192.168.202.132@tcp130 [14270.122345] LNet: Removed LNI 192.168.202.132@tcp131 [14270.133654] LNet: Removed LNI 192.168.202.132@tcp132 [14270.135597] LNet: Removed LNI 192.168.202.132@tcp133 [14270.144519] LNet: Removed LNI 192.168.202.132@tcp134 [14270.173059] LNet: Removed LNI 192.168.202.132@tcp135 [14271.367449] Key type .llcrypt unregistered [14271.373219] Key type ._llcrypt unregistered [14297.529205] Key type ._llcrypt registered [14297.531740] Key type .llcrypt registered [14297.693119] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [14315.959943] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [14318.019600] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14318.126686] alg: No test for adler32 (adler32-zlib) [14319.487623] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [14319.845702] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [14321.591217] Key type lgssc registered [14323.048575] Lustre: Echo OBD driver; http://www.lustre.org/ [14363.918248] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 08:48:58 (1789649338) [14383.788854] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [14386.369533] Key type lgssc unregistered [14386.665653] LNet: 471641:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14386.674750] LNetError: 471641:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14386.707295] LNet: Removed LNI 192.168.202.132@tcp [14387.398779] Key type .llcrypt unregistered [14387.400992] Key type ._llcrypt unregistered [14403.513740] Key type ._llcrypt registered [14403.515919] Key type .llcrypt registered [14403.622761] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [14405.215379] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14405.515545] alg: No test for adler32 (adler32-zlib) [14406.767626] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [14407.168322] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [14408.927376] Key type lgssc registered [14410.042417] Lustre: Echo OBD driver; http://www.lustre.org/ [14422.215480] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14422.238684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14423.534535] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14423.552737] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14423.604708] Lustre: lustre-MDT0000: new disk, initializing [14423.679591] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14423.696727] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14430.225267] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [14441.672170] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14441.743889] Lustre: 472850:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14441.781336] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14441.784447] Lustre: Skipped 1 previous similar message [14441.842685] Lustre: lustre-MDT0001: new disk, initializing [14441.897914] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14441.919864] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14441.928501] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14448.714510] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [14461.215464] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14466.815098] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14473.286647] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14473.572292] Lustre: lustre-OST0000: new disk, initializing [14473.576446] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14473.590293] Lustre: 474817:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14473.718580] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14480.039728] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [14483.521044] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14483.533114] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14483.683703] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14490.021303] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14494.800808] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing lnet_if_list [14500.669882] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14500.704558] LNet: Added LNI 192.168.202.132@tcp10 [8/256/0/180] [14507.381328] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14507.412975] LNet: Added LNI 192.168.202.132@tcp11 [8/256/0/180] [14512.375605] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14512.426607] LNet: Added LNI 192.168.202.132@tcp12 [8/256/0/180] [14518.134631] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14518.179827] LNet: Added LNI 192.168.202.132@tcp13 [8/256/0/180] [14524.265332] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14524.307210] LNet: Added LNI 192.168.202.132@tcp14 [8/256/0/180] [14530.281464] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14530.322662] LNet: Added LNI 192.168.202.132@tcp15 [8/256/0/180] [14536.248559] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [14536.300294] LNet: Added LNI 192.168.202.132@tcp16 [8/256/0/180] [14544.052782] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [14544.113140] LNet: Added LNI 192.168.202.132@tcp17 [8/256/0/180] [14552.420369] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [14552.501672] LNet: Added LNI 192.168.202.132@tcp18 [8/256/0/180] [14559.071646] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [14559.099741] LNet: Added LNI 192.168.202.132@tcp19 [8/256/0/180] [14566.322718] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [14566.341901] LNet: Added LNI 192.168.202.132@tcp20 [8/256/0/180] [14572.122775] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [14572.143044] LNet: Added LNI 192.168.202.132@tcp21 [8/256/0/180] [14577.372514] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [14577.445941] LNet: Added LNI 192.168.202.132@tcp22 [8/256/0/180] [14584.015432] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [14584.037863] LNet: Added LNI 192.168.202.132@tcp23 [8/256/0/180] [14590.532373] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [14590.571729] LNet: Added LNI 192.168.202.132@tcp24 [8/256/0/180] [14597.055875] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [14597.095167] LNet: Added LNI 192.168.202.132@tcp25 [8/256/0/180] [14603.919555] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [14603.962461] LNet: Added LNI 192.168.202.132@tcp26 [8/256/0/180] [14610.569356] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [14610.600173] LNet: Added LNI 192.168.202.132@tcp27 [8/256/0/180] [14616.710890] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [14616.772175] LNet: Added LNI 192.168.202.132@tcp28 [8/256/0/180] [14624.567981] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [14624.632750] LNet: Added LNI 192.168.202.132@tcp29 [8/256/0/180] [14634.646821] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [14634.765397] LNet: Added LNI 192.168.202.132@tcp30 [8/256/0/180] [14643.968781] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [14643.992366] LNet: Added LNI 192.168.202.132@tcp31 [8/256/0/180] [14651.361790] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [14651.388281] LNet: Added LNI 192.168.202.132@tcp32 [8/256/0/180] [14660.468501] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [14660.517517] LNet: Added LNI 192.168.202.132@tcp33 [8/256/0/180] [14669.186376] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [14669.233585] LNet: Added LNI 192.168.202.132@tcp34 [8/256/0/180] [14680.660156] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [14680.729144] LNet: Added LNI 192.168.202.132@tcp35 [8/256/0/180] [14690.157706] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [14690.223068] LNet: Added LNI 192.168.202.132@tcp36 [8/256/0/180] [14701.841609] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [14701.881323] LNet: Added LNI 192.168.202.132@tcp37 [8/256/0/180] [14710.002465] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [14710.066624] LNet: Added LNI 192.168.202.132@tcp38 [8/256/0/180] [14717.558218] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [14717.610959] LNet: Added LNI 192.168.202.132@tcp39 [8/256/0/180] [14726.367591] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [14726.386232] LNet: Added LNI 192.168.202.132@tcp40 [8/256/0/180] [14732.154872] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [14732.173475] LNet: Added LNI 192.168.202.132@tcp41 [8/256/0/180] [14740.182449] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [14740.228124] LNet: Added LNI 192.168.202.132@tcp42 [8/256/0/180] [14750.174433] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [14750.224242] LNet: Added LNI 192.168.202.132@tcp43 [8/256/0/180] [14761.117184] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [14761.153536] LNet: Added LNI 192.168.202.132@tcp44 [8/256/0/180] [14772.417462] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [14772.503827] LNet: Added LNI 192.168.202.132@tcp45 [8/256/0/180] [14787.331325] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [14787.390288] LNet: Added LNI 192.168.202.132@tcp46 [8/256/0/180] [14800.666966] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [14800.723035] LNet: Added LNI 192.168.202.132@tcp47 [8/256/0/180] [14810.076671] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [14810.139748] LNet: Added LNI 192.168.202.132@tcp48 [8/256/0/180] [14823.415599] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [14823.438465] LNet: Added LNI 192.168.202.132@tcp49 [8/256/0/180] [14834.785632] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [14834.827692] LNet: Added LNI 192.168.202.132@tcp50 [8/256/0/180] [14861.792606] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14861.801280] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14861.824571] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14862.839589] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14862.841266] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14862.872757] Lustre: Skipped 1 previous similar message [14864.939140] LustreError: 497955:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x0:0x0].0x0 (0000000089d1073a) refcount nonzero (1) after lock cleanup; forcing cleanup. [14865.586821] Lustre: server umount lustre-MDT0000 complete [14871.383962] LustreError: 472842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789649847 with bad export cookie 15697497444425195458 [14871.403338] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14871.417410] LustreError: 472842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14872.133528] Lustre: server umount lustre-MDT0001 complete [14889.449297] Lustre: 472123:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789649849/real 1789649849] req@00000000d9ac55e5 x1876583391622144/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789649865 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14889.478162] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14892.511213] Lustre: 472120:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789649852/real 1789649852] req@0000000019e8cd68 x1876583391622272/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789649868 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14894.559248] Lustre: 472121:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789649854/real 1789649854] req@00000000e774ea7d x1876583391622400/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789649870 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14894.888453] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [14897.631367] Lustre: 472123:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789649857/real 1789649857] req@00000000ed8c35ee x1876583391622656/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789649873 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14901.733401] Lustre: 472122:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789649862/real 1789649862] req@0000000064e669bb x1876583391622784/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789649878 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14906.111595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14916.622887] LustreError: 474811:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14916.652377] LustreError: 474811:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [14916.741559] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14917.728931] LustreError: 474812:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14917.767140] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14919.781619] LustreError: 474811:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14922.161625] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [14925.290502] LustreError: 474812:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14930.423263] LustreError: 474811:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14933.157276] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14933.854143] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14934.837536] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14940.733956] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [14951.460804] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14960.810181] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14988.255950] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14988.264514] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14988.277257] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14988.282569] Lustre: Skipped 2 previous similar messages [14990.320711] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14990.326415] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14996.451443] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14996.468757] Lustre: Skipped 3 previous similar messages [15001.058049] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15001.633318] Lustre: server umount lustre-OST0000 complete [15010.785318] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15010.800689] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15010.821759] Lustre: Skipped 1 previous similar message [15010.826208] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15013.752833] Lustre: server umount lustre-MDT0000 complete [15016.928483] LustreError: 499101:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15016.959969] LustreError: 499101:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [15017.952042] Lustre: 472123:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789649977/real 1789649977] req@00000000832b9a64 x1876583391659648/t0(0) o400->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 224/224 e 0 to 1 dl 1789649993 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15018.295087] LustreError: 478597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789649994 with bad export cookie 15697497444425198020 [15018.301231] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15018.305647] LustreError: 478597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [15018.809285] Lustre: server umount lustre-MDT0001 complete [15034.164737] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [15037.844603] Key type lgssc unregistered [15038.478591] LNet: 501901:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15038.487597] LNetError: 501901:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15038.497079] LNet: Removed LNI 192.168.202.132@tcp [15038.499025] LNet: Removed LNI 192.168.202.132@tcp10 [15038.501270] LNet: Removed LNI 192.168.202.132@tcp11 [15038.503455] LNet: Removed LNI 192.168.202.132@tcp12 [15038.509813] LNet: Removed LNI 192.168.202.132@tcp13 [15038.514854] LNet: Removed LNI 192.168.202.132@tcp14 [15038.517436] LNet: Removed LNI 192.168.202.132@tcp15 [15038.522845] LNet: Removed LNI 192.168.202.132@tcp16 [15038.527503] LNet: Removed LNI 192.168.202.132@tcp17 [15038.536769] LNet: Removed LNI 192.168.202.132@tcp18 [15038.539452] LNet: Removed LNI 192.168.202.132@tcp19 [15038.544370] LNet: Removed LNI 192.168.202.132@tcp20 [15038.546430] LNet: Removed LNI 192.168.202.132@tcp21 [15038.548571] LNet: Removed LNI 192.168.202.132@tcp22 [15038.551142] LNet: Removed LNI 192.168.202.132@tcp23 [15038.553163] LNet: Removed LNI 192.168.202.132@tcp24 [15038.554721] LNet: Removed LNI 192.168.202.132@tcp25 [15038.556030] LNet: Removed LNI 192.168.202.132@tcp26 [15038.557584] LNet: Removed LNI 192.168.202.132@tcp27 [15038.561299] LNet: Removed LNI 192.168.202.132@tcp28 [15038.563727] LNet: Removed LNI 192.168.202.132@tcp29 [15038.565844] LNet: Removed LNI 192.168.202.132@tcp30 [15038.568161] LNet: Removed LNI 192.168.202.132@tcp31 [15038.569703] LNet: Removed LNI 192.168.202.132@tcp32 [15038.571712] LNet: Removed LNI 192.168.202.132@tcp33 [15038.574191] LNet: Removed LNI 192.168.202.132@tcp34 [15038.576467] LNet: Removed LNI 192.168.202.132@tcp35 [15038.578584] LNet: Removed LNI 192.168.202.132@tcp36 [15038.580433] LNet: Removed LNI 192.168.202.132@tcp37 [15038.582748] LNet: Removed LNI 192.168.202.132@tcp38 [15038.584874] LNet: Removed LNI 192.168.202.132@tcp39 [15038.586917] LNet: Removed LNI 192.168.202.132@tcp40 [15038.588880] LNet: Removed LNI 192.168.202.132@tcp41 [15038.590776] LNet: Removed LNI 192.168.202.132@tcp42 [15038.592933] LNet: Removed LNI 192.168.202.132@tcp43 [15038.595137] LNet: Removed LNI 192.168.202.132@tcp44 [15038.597181] LNet: Removed LNI 192.168.202.132@tcp45 [15038.599731] LNet: Removed LNI 192.168.202.132@tcp46 [15038.602231] LNet: Removed LNI 192.168.202.132@tcp47 [15038.604755] LNet: Removed LNI 192.168.202.132@tcp48 [15038.607116] LNet: Removed LNI 192.168.202.132@tcp49 [15038.626578] LNet: Removed LNI 192.168.202.132@tcp50 [15039.684613] Key type .llcrypt unregistered [15039.687294] Key type ._llcrypt unregistered [15063.974940] Key type ._llcrypt registered [15063.977541] Key type .llcrypt registered [15064.151751] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [15086.773498] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [15088.715769] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15088.775147] alg: No test for adler32 (adler32-zlib) [15090.209648] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [15090.613777] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [15092.560508] Key type lgssc registered [15095.013232] Lustre: Echo OBD driver; http://www.lustre.org/ [15140.482492] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15140.576182] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15141.940499] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15142.013206] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15142.150560] Lustre: lustre-MDT0000: new disk, initializing [15142.267272] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15142.288758] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15149.468875] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [15167.053990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15167.331963] Lustre: 505105:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15167.474615] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15167.487142] Lustre: Skipped 1 previous similar message [15167.665580] Lustre: lustre-MDT0001: new disk, initializing [15167.968994] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15168.060893] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15168.084283] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15174.749605] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [15190.107144] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15195.755482] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15204.966019] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15205.153299] Lustre: *** cfs_fail_loc=90f, val=0*** [15205.260767] Lustre: lustre-OST0000: new disk, initializing [15205.267242] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15205.287252] Lustre: 507125:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15205.420741] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15212.330581] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [15213.164945] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15213.187479] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15213.360753] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15225.360364] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15234.022724] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15234.034334] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15234.039271] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15234.061083] Lustre: Skipped 1 previous similar message [15239.137461] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15239.161349] Lustre: Skipped 1 previous similar message [15240.012152] Lustre: server umount lustre-OST0000 complete [15249.382881] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15249.399892] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15249.402842] Lustre: Skipped 2 previous similar messages [15249.421212] Lustre: Skipped 1 previous similar message [15252.350274] Lustre: server umount lustre-MDT0000 complete [15256.997739] LustreError: 508078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789650233 with bad export cookie 1390355473923652220 [15256.998926] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15257.006911] LustreError: 508078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15257.511363] Lustre: server umount lustre-MDT0001 complete [15271.017386] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [15274.682951] Key type lgssc unregistered [15275.180216] LNet: 509035:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15275.190195] LNetError: 509035:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15275.207534] LNet: Removed LNI 192.168.202.132@tcp [15276.322689] Key type .llcrypt unregistered [15276.328741] Key type ._llcrypt unregistered [15302.548242] Key type ._llcrypt registered [15302.553495] Key type .llcrypt registered [15302.730820] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [15306.126626] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15306.273329] alg: No test for adler32 (adler32-zlib) [15307.448929] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [15307.760782] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [15309.503472] Key type lgssc registered [15311.065850] Lustre: Echo OBD driver; http://www.lustre.org/ [15320.625528] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15320.656533] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15322.160047] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15327.053171] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [15332.327195] LustreError: 510302:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15332.370433] LustreError: 510302:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [15337.443115] LustreError: 510303:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15338.159716] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15339.093056] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15345.198467] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [15356.637626] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15365.574643] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15373.745506] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15374.374186] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15385.221808] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [15397.733663] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15410.017025] loop0: detected capacity change from 0 to 204800000 [15410.335546] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15410.374324] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15410.413529] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15418.950691] loop0: detected capacity change from 0 to 204800000 [15419.171034] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15419.194200] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15419.204223] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15426.874880] loop0: detected capacity change from 0 to 204800000 [15433.082530] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15433.365257] Lustre: fs15246-OST0000: new disk, initializing [15433.369424] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [15433.377265] Lustre: 513900:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15433.509553] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15440.494926] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [15450.682088] loop1: detected capacity change from 0 to 204800000 [15456.942251] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15457.196767] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [15457.274512] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [15457.398904] Lustre: fs15246-MDT0000: new disk, initializing [15457.675242] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15457.720249] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15457.868580] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [15458.917927] Lustre: 515022:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000078664a08 x1876584336250496/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1789650451 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [15463.021425] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [15468.071194] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [15468.081615] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [15468.130356] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15492.080168] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15492.117527] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15494.316374] Lustre: server umount fs15246-MDT0000 complete [15497.214095] LustreError: 510297:0:(ldlm_lib.c:1199:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15502.313473] LustreError: 511836:0:(ldlm_lib.c:1199:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15506.111923] Lustre: server umount fs15246-OST0000 complete [15523.299257] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15523.333363] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15523.354815] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15525.345419] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15525.370390] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15525.393868] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15527.906527] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15529.937746] Lustre: server umount lustre-OST0000 complete [15538.145962] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15538.173578] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15538.191328] Lustre: Skipped 2 previous similar messages [15543.209146] Lustre: server umount lustre-MDT0000 complete [15543.268123] LustreError: 510302:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15543.281679] LustreError: 510302:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [15548.257256] LustreError: 510284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789650524 with bad export cookie 1520586387307854067 [15548.260378] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15548.280991] LustreError: 510284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [15548.638920] Lustre: server umount lustre-MDT0001 complete [15560.435134] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [15563.008193] Key type lgssc unregistered [15563.443464] LNet: 518124:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15563.450727] LNetError: 518124:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15563.478660] LNet: Removed LNI 192.168.202.132@tcp [15564.255300] Key type .llcrypt unregistered [15564.257653] Key type ._llcrypt unregistered [15586.178202] Key type ._llcrypt registered [15586.180770] Key type .llcrypt registered [15586.316895] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [15587.405860] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15587.446117] alg: No test for adler32 (adler32-zlib) [15588.658595] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [15588.979682] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [15590.711346] Key type lgssc registered [15591.906973] Lustre: Echo OBD driver; http://www.lustre.org/ [15636.991808] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [15647.246148] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [15689.491594] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15689.517540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15691.013616] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15691.070599] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15691.270388] Lustre: lustre-MDT0000: new disk, initializing [15691.513089] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15691.550948] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15696.523142] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [15712.199335] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15712.384668] Lustre: 522590:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15712.492734] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15712.495732] Lustre: Skipped 1 previous similar message [15712.688068] Lustre: lustre-MDT0001: new disk, initializing [15712.885842] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15712.964251] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15712.983604] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15718.902563] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [15730.250389] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15736.429610] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15743.863675] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15744.173512] Lustre: lustre-OST0000: new disk, initializing [15744.177630] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15744.184223] Lustre: 524558:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15744.301284] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15750.461340] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [15753.827913] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15753.848066] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15754.015418] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15761.193980] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15767.488240] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15767.754697] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15774.145851] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15774.574842] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15779.814426] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15779.823436] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15779.841105] Lustre: Skipped 1 previous similar message [15783.263742] Lustre: server umount lustre-OST0000 complete [15789.537192] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15789.542689] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15789.564991] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15789.567525] Lustre: Skipped 1 previous similar message [15793.912345] Lustre: server umount lustre-MDT0000 complete [15798.273341] LustreError: 523385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789650774 with bad export cookie 15768810853945419372 [15798.290110] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15798.291875] LustreError: 523385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15798.593499] Lustre: server umount lustre-MDT0001 complete [15827.824144] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [15840.238848] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [15875.190252] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 15546 sec ========= 09:14:10 (1789650850) [15877.126806] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 09:14:12 (1789650852) === [15880.371474] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 09:14:15 (1789650855) === [15914.917278] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [15917.758693] Key type lgssc unregistered [15918.054172] LNet: 531206:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15918.060024] LNetError: 531206:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15918.075329] LNet: Removed LNI 192.168.202.132@tcp [15918.920857] Key type .llcrypt unregistered [15918.923100] Key type ._llcrypt unregistered