[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 468681729 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001023] APIC: Switch to symmetric I/O mode setup [ 0.003325] x2apic enabled [ 0.004006] Switched APIC routing to physical x2apic. [ 0.005012] kvm-guest: setup PV IPIs [ 0.008391] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009018] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010009] pid_max: default: 32768 minimum: 301 [ 0.012074] LSM: Security Framework initializing [ 0.013041] Yama: becoming mindful. [ 0.014030] SELinux: Initializing. [ 0.015058] *** VALIDATE selinux *** [ 0.022292] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027202] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028128] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029083] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031086] *** VALIDATE tmpfs *** [ 0.032387] *** VALIDATE proc *** [ 0.033187] *** VALIDATE cgroup *** [ 0.034006] *** VALIDATE cgroup2 *** [ 0.035210] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036123] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037004] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038091] Spectre V2 : User space: Vulnerable [ 0.039006] Speculative Store Bypass: Vulnerable [ 0.042297] debug: unmapping init [mem 0xffffffffa1a59000-0xffffffffa1a60fff] [ 0.044144] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045574] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046016] ... version: 2 [ 0.047006] ... bit width: 48 [ 0.048006] ... generic registers: 4 [ 0.049006] ... value mask: 0000ffffffffffff [ 0.050006] ... max period: 00007fffffffffff [ 0.051008] ... fixed-purpose events: 3 [ 0.052006] ... event mask: 000000070000000f [ 0.053272] rcu: Hierarchical SRCU implementation. [ 0.055259] smp: Bringing up secondary CPUs ... [ 0.056450] x86: Booting SMP configuration: [ 0.057019] .... node #0, CPUs: #1 #2 #3 [ 0.060012] smp: Brought up 1 node, 4 CPUs [ 0.061007] smpboot: Max logical packages: 1 [ 0.062009] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.127209] node 0 deferred pages initialised in 63ms [ 0.130101] devtmpfs: initialized [ 0.132318] x86/mm: Memory block size: 128MB [ 0.136095] gcov: version magic: 0x41383552 [ 0.139238] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.140066] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.141000] pinctrl core: initialized pinctrl subsystem [ 0.143131] [ 0.143562] ************************************************************* [ 0.145006] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.147008] ** ** [ 0.148006] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.150007] ** ** [ 0.152007] ** This means that this kernel is built to expose internal ** [ 0.154008] ** IOMMU data structures, which may compromise security on ** [ 0.156008] ** your system. ** [ 0.158008] ** ** [ 0.160007] ** If you see this message and you are not debugging the ** [ 0.162007] ** kernel, report this immediately to your vendor! ** [ 0.163006] ** ** [ 0.165007] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.168007] ************************************************************* [ 0.170710] NET: Registered protocol family 16 [ 0.171403] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.174040] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.176038] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.180027] cpuidle: using governor menu [ 0.181723] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.183406] PCI: Using configuration type 1 for base access [ 0.185095] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.192093] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.194012] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.198039] cryptd: max_cpu_qlen set to 1000 [ 0.200193] ACPI: Added _OSI(Module Device) [ 0.201022] ACPI: Added _OSI(Processor Device) [ 0.203008] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.204007] ACPI: Added _OSI(Processor Aggregator Device) [ 0.208491] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.215743] ACPI: Interpreter enabled [ 0.217044] ACPI: PM: (supports S0 S3 S4 S5) [ 0.218009] ACPI: Using IOAPIC for interrupt routing [ 0.220103] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.223284] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.233301] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.235020] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.237015] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.240062] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.245765] acpiphp: Slot [2] registered [ 0.246082] acpiphp: Slot [5] registered [ 0.248051] acpiphp: Slot [6] registered [ 0.249072] acpiphp: Slot [7] registered [ 0.250088] acpiphp: Slot [8] registered [ 0.251153] acpiphp: Slot [9] registered [ 0.253085] acpiphp: Slot [10] registered [ 0.255079] acpiphp: Slot [3] registered [ 0.256070] acpiphp: Slot [4] registered [ 0.257148] acpiphp: Slot [11] registered [ 0.259071] acpiphp: Slot [12] registered [ 0.260059] acpiphp: Slot [13] registered [ 0.262089] acpiphp: Slot [14] registered [ 0.263081] acpiphp: Slot [15] registered [ 0.265077] acpiphp: Slot [16] registered [ 0.266079] acpiphp: Slot [17] registered [ 0.267061] acpiphp: Slot [18] registered [ 0.269048] acpiphp: Slot [19] registered [ 0.270059] acpiphp: Slot [20] registered [ 0.271076] acpiphp: Slot [21] registered [ 0.272073] acpiphp: Slot [22] registered [ 0.273056] acpiphp: Slot [23] registered [ 0.274116] acpiphp: Slot [24] registered [ 0.276087] acpiphp: Slot [25] registered [ 0.277095] acpiphp: Slot [26] registered [ 0.279074] acpiphp: Slot [27] registered [ 0.281085] acpiphp: Slot [28] registered [ 0.283074] acpiphp: Slot [29] registered [ 0.284173] acpiphp: Slot [30] registered [ 0.287084] acpiphp: Slot [31] registered [ 0.288043] PCI host bridge to bus 0000:00 [ 0.290015] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.291011] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.293013] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.295012] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.297015] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.299013] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.300137] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.302843] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.305034] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.314013] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.318046] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.320017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.322011] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.324008] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.325640] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.330205] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.332035] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.334742] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.339009] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.351892] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.357050] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.364219] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.379013] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.398012] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.461034] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.486946] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.502012] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.514022] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.555012] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.572027] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.595015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.614011] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.680014] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.709065] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.724013] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.738012] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.783013] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.798567] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.813013] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.823012] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.873014] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.898444] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.914013] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.929012] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.947015] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.957111] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.959277] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.961263] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.964324] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.965136] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.970048] iommu: Default domain type: Passthrough [ 0.971272] SCSI subsystem initialized [ 0.975141] ACPI: bus type USB registered [ 0.976068] usbcore: registered new interface driver usbfs [ 0.978057] usbcore: registered new interface driver hub [ 0.979077] usbcore: registered new device driver usb [ 0.980118] pps_core: LinuxPPS API ver. 1 registered [ 0.981000] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.983044] PTP clock support registered [ 0.984333] EDAC MC: Ver: 3.0.0 [ 0.986453] PCI: Using ACPI for IRQ routing [ 0.987000] NetLabel: Initializing [ 0.988008] NetLabel: domain hash size = 128 [ 0.989005] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.991063] NetLabel: unlabeled traffic allowed by default [ 0.993217] vgaarb: loaded [ 0.994632] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.996010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.002156] clocksource: Switched to clocksource kvm-clock [ 1.116734] VFS: Disk quotas dquot_6.6.0 [ 1.118217] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.120825] *** VALIDATE ramfs *** [ 1.122113] *** VALIDATE hugetlbfs *** [ 1.123455] pnp: PnP ACPI init [ 1.125956] pnp: PnP ACPI: found 6 devices [ 1.142265] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.145312] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.147371] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.149404] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.151765] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.154098] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.156738] NET: Registered protocol family 2 [ 1.159064] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.163873] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.167098] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.172158] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.175349] TCP: Hash tables configured (established 65536 bind 65536) [ 1.178043] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.180809] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.184153] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.186962] NET: Registered protocol family 1 [ 1.189490] RPC: Registered named UNIX socket transport module. [ 1.194552] RPC: Registered udp transport module. [ 1.196250] RPC: Registered tcp transport module. [ 1.197963] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.200251] NET: Registered protocol family 44 [ 1.201829] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.203859] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.205844] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.207996] PCI: CLS 0 bytes, default 64 [ 1.209534] Unpacking initramfs... [ 2.611150] debug: unmapping init [mem 0xffff9718fcc54000-0xffff9718fffbffff] [ 2.615688] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.617640] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.621904] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.132355] Initialise system trusted keyrings [ 3.133824] Key type blacklist registered [ 3.135761] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.143534] zbud: loaded [ 3.145865] *** VALIDATE nfs *** [ 3.146988] *** VALIDATE nfs4 *** [ 3.148273] pstore: using deflate compression [ 3.151255] Platform Keyring initialized [ 3.272515] NET: Registered protocol family 38 [ 3.273905] Key type asymmetric registered [ 3.275133] Asymmetric key parser 'x509' registered [ 3.276558] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.279547] io scheduler mq-deadline registered [ 3.280922] io scheduler kyber registered [ 3.282313] io scheduler bfq registered [ 3.284537] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.287349] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.290220] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.293192] ACPI: Power Button [PWRF] [ 3.298619] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.305166] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.331660] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.349728] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.403945] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.432353] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.463198] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.469043] Non-volatile memory driver v1.3 [ 3.470516] Linux agpgart interface v0.103 [ 3.502162] virtio_blk virtio1: [vda] 145888 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.505108] vda: detected capacity change from 0 to 74694656 [ 3.529995] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.532879] vdb: detected capacity change from 0 to 1073741824 [ 3.556166] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.559275] vdc: detected capacity change from 0 to 2621440000 [ 3.587586] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.590330] vdd: detected capacity change from 0 to 2621440000 [ 3.608708] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.612037] vde: detected capacity change from 0 to 4294967296 [ 3.633215] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.636119] vdf: detected capacity change from 0 to 4294967296 [ 3.646235] libphy: Fixed MDIO Bus: probed [ 3.668830] usbcore: registered new interface driver usbserial_generic [ 3.671208] usbserial: USB Serial support registered for generic [ 3.673300] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.678617] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.680480] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.683186] mousedev: PS/2 mouse device common for all mice [ 3.686610] rtc_cmos 00:05: RTC can wake from S4 [ 3.690269] rtc_cmos 00:05: registered as rtc0 [ 3.692584] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.695413] intel_pstate: CPU model not supported [ 3.695632] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.702121] hid: raw HID events driver (C) Jiri Kosina [ 3.704260] usbcore: registered new interface driver usbhid [ 3.706392] usbhid: USB HID core driver [ 3.708103] drop_monitor: Initializing network drop monitor service [ 3.710533] Initializing XFRM netlink socket [ 3.712954] NET: Registered protocol family 10 [ 3.713708] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.717593] Segment Routing with IPv6 [ 3.721432] NET: Registered protocol family 17 [ 3.724915] mpls_gso: MPLS GSO support [ 3.730351] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.735697] RAS: Correctable Errors collector initialized. [ 3.738102] AVX version of gcm_enc/dec engaged. [ 3.739429] AES CTR mode by8 optimization enabled [ 3.838051] sched_clock: Marking stable (3838024555, 0)->(4708452311, -870427756) [ 3.842258] registered taskstats version 1 [ 3.845205] Loading compiled-in X.509 certificates [ 3.847032] zswap: loaded using pool lzo/zbud [ 3.875184] Key type big_key registered [ 3.889048] Key type encrypted registered [ 3.890597] ima: No TPM chip found, activating TPM-bypass! [ 3.892849] ima: Allocated hash algorithm: sha1 [ 3.894728] ima: No architecture policies found [ 3.896468] evm: Initialising EVM extended attributes: [ 3.898405] evm: security.selinux [ 3.899734] evm: security.ima [ 3.900736] evm: security.capability [ 3.902277] evm: HMAC attrs: 0x1 [ 3.905728] rtc_cmos 00:05: setting system clock to 2026-08-05 12:41:44 UTC (1785933704) [ 3.911928] debug: unmapping init [mem 0xffffffffa2a03000-0xffffffffa2bfffff] [ 3.915510] debug: unmapping init [mem 0xffffffffa1782000-0xffffffffa1a58fff] [ 3.923091] Write protecting the kernel read-only data: 28672k [ 3.927161] debug: unmapping init [mem 0xffffffff9fe03000-0xffffffff9fffffff] [ 3.930978] debug: unmapping init [mem 0xffffffffa0714000-0xffffffffa07fffff] [ 3.964566] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.973681] systemd[1]: Detected virtualization kvm. [ 3.975405] systemd[1]: Detected architecture x86-64. [ 3.977249] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.001712] systemd[1]: No hostname configured. [ 4.003380] systemd[1]: Set hostname to . [ 4.005381] random: systemd: uninitialized urandom read (16 bytes read) [ 4.007852] systemd[1]: Initializing machine ID from random generator. [ 4.046705] random: ln: uninitialized urandom read (6 bytes read) [ 4.125128] random: systemd: uninitialized urandom read (16 bytes read) [ 4.127584] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 4.132490] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 4.135630] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Slices. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.681904] device-mapper: uevent: version 1.0.3 [ 4.683881] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.369984] virtio_net virtio0 ens2: renamed from eth0 [ 5.381327] random: fast init done [ 5.432704] scsi host0: ata_piix [ 5.451166] scsi host1: ata_piix [ 5.452948] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.455418] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.163305] dracut-initqueue[589]: RTNETLINK answers: File exists [ 10.419496] random: crng init done [ 10.420659] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 11.743382] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 14.873398] printk: systemd: 26 output lines suppressed due to ratelimiting [ 15.780767] SELinux: Disabled at runtime. [ 16.010456] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 16.029866] systemd[1]: Detected virtualization kvm. [ 16.039934] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 17.587433] systemd[1]: initrd-switch-root.service: Succeeded. [ 17.602714] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 17.623702] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 17.633441] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 17.640815] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 17.660900] systemd[1]: Starting Journal Service... Starting Journal Service... [ 17.680760] systemd[1]: proc-sys-fs-binfmt_misc.automount: Refusing to start, unit to trigger not loaded. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ 17.792770] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Create list of required st…ce nodes for the current kernel... Mounting Kernel Debug File System... [ OK ] Created slice system-getty.slice. Mounting Huge Pages File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. Mounting POSIX Message Queue File System... Starting Apply Kernel Variables... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Remount Root and Kernel File Systems... [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 19.596713] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 21.002931] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 21.099897] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 21.639953] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 21.723915] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit)[ 26.872059] Key type dns_resolver registered [*** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit)[ 28.180357] NFS: Registering the id_resolver key type [ 28.190923] Key type id_resolver registered [ 28.194715] Key type id_legacy registered [ *** ] A start job is running for Configur…only root support (10s / no limit) [ ***] A start job is running for Configur…only root support (11s / no limit) [ **] A start job is running for Configur…only root support (11s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg320-server login: [ 98.491054] hrtimer: interrupt took 3667856 ns [ 123.313365] libcfs: loading out-of-tree module taints kernel. [ 123.493478] Key type ._llcrypt registered [ 123.496322] Key type .llcrypt registered [ 123.657857] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [ 162.304993] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 165.785556] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 165.813157] alg: No test for adler32 (adler32-zlib) [ 167.796722] Lustre: Lustre: Build Version: 2.17.56_50_g82e980f [ 169.340412] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [ 171.167154] Key type lgssc registered [ 172.914522] Lustre: Echo OBD driver; http://www.lustre.org/ [ 194.836776] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 248.309506] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 268.393030] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 268.488328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 269.783944] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 269.807208] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 269.867836] Lustre: lustre-MDT0000: new disk, initializing [ 269.919799] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 269.933809] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 276.324640] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 294.809488] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 294.972543] Lustre: 6533:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 295.080745] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 295.088568] Lustre: Skipped 1 previous similar message [ 295.272662] Lustre: lustre-MDT0001: new disk, initializing [ 295.369565] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 295.416189] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 295.428236] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 302.897872] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 310.169944] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 324.801276] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 325.292220] Lustre: lustre-OST0000: new disk, initializing [ 325.306413] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 325.321164] Lustre: 8472:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 325.446451] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 332.341099] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 332.350259] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 332.443994] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 333.960535] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 356.285495] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 356.674880] Lustre: lustre-OST0001: new disk, initializing [ 356.684717] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 356.705872] Lustre: 9551:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 356.843831] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 363.601301] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 363.640138] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 363.774347] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 367.258898] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 382.869639] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 391.207969] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 400.346214] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing check_logdir /tmp/testlogs/ [ 407.167980] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing yml_node [ 415.349253] Lustre: DEBUG MARKER: Client: 2.17.56.50 [ 419.595898] Lustre: DEBUG MARKER: MDS: 2.17.56.50 [ 422.132515] Lustre: DEBUG MARKER: OSS: 2.17.56.50 [ 424.899884] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Aug 5 08:48:42 EDT 2026 [ 449.406690] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 451.308582] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 474.591755] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 474.605351] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 474.618860] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 476.649166] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 476.659927] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 476.696443] Lustre: Skipped 1 previous similar message [ 481.764274] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 481.771940] Lustre: Skipped 6 previous similar messages [ 486.367230] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 486.864443] Lustre: server umount lustre-MDT0000 complete [ 492.002186] LustreError: 6545:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 492.038765] LustreError: 6545:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 493.026154] LustreError: 6525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785934193 with bad export cookie 18126127861988516371 [ 493.030700] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 493.059897] LustreError: 6525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 497.135022] LustreError: 6539:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 497.136916] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 497.147352] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 497.160461] LustreError: 6539:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 497.191407] Lustre: Skipped 2 previous similar messages [ 498.153292] LustreError: 12184:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 498.170728] LustreError: 12184:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 499.828535] Lustre: server umount lustre-MDT0001 complete [ 505.159409] Lustre: server umount lustre-OST0000 complete [ 511.309022] Lustre: server umount lustre-OST0001 complete [ 521.961090] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [ 540.022228] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 588.879506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 589.305948] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 589.398462] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 589.604578] Lustre: lustre-MDT0000: new disk, initializing [ 589.737925] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 589.785032] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 596.550742] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 615.771651] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 615.963036] Lustre: 15130:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 616.026474] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 616.029367] Lustre: Skipped 1 previous similar message [ 616.125597] Lustre: lustre-MDT0001: new disk, initializing [ 616.325050] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 616.403779] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 616.414548] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 622.702863] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 634.974611] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 642.066767] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 651.846661] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 652.438235] Lustre: lustre-OST0000: new disk, initializing [ 652.454471] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 652.464347] Lustre: 17101:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 652.634434] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 654.038764] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 654.053629] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 654.275207] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 662.726379] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 677.128857] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 684.325137] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 684.608659] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 691.855225] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 692.174107] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 695.265190] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 695.268485] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 695.284506] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 695.284512] Lustre: Skipped 3 previous similar messages [ 695.334968] Lustre: Skipped 1 previous similar message [ 700.385615] LustreError: 17086:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 700.428118] LustreError: 17086:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 700.799690] Lustre: server umount lustre-OST0000 complete [ 708.580742] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 708.599726] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 708.633919] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 708.636607] Lustre: Skipped 1 previous similar message [ 714.342637] Lustre: server umount lustre-MDT0000 complete [ 715.747492] LustreError: 16008:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 715.771917] LustreError: 16008:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 718.788406] LustreError: 15122:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785934419 with bad export cookie 18126127861988517904 [ 718.792369] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 718.802903] LustreError: 15122:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 719.617588] Lustre: server umount lustre-MDT0001 complete [ 726.298819] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 08:53:44 (1785934424) [ 729.651894] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 732.091245] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 08:53:50 (1785934430) [ 740.854101] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 741.214060] LustreError: 19515:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 741.274841] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 746.706742] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 758.101215] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 765.557496] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 776.836906] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 785.401513] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 796.449091] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 807.264300] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 818.051125] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 829.920519] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 829.939844] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 829.955769] Lustre: Skipped 1 previous similar message [ 829.968500] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 829.986619] Lustre: Skipped 2 previous similar messages [ 836.095691] LustreError: 21340:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 836.119813] LustreError: 21340:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 836.250466] Lustre: server umount lustre-OST0000 complete [ 845.797661] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 845.809862] LustreError: Skipped 1 previous similar message [ 846.339070] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 846.383171] Lustre: Skipped 2 previous similar messages [ 848.223801] Lustre: server umount lustre-MDT0000 complete [ 853.976991] LustreError: 19497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785934554 with bad export cookie 18126127861988518786 [ 853.999876] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 854.001341] LustreError: 19497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 854.710376] Lustre: server umount lustre-MDT0001 complete [ 871.003760] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [ 875.278889] Key type lgssc unregistered [ 875.785742] LNet: 23059:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 875.792691] LNetError: 23059:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 875.841219] LNet: Removed LNI 192.168.203.120@tcp [ 877.516184] Key type .llcrypt unregistered [ 877.518189] Key type ._llcrypt unregistered [ 910.619769] Key type ._llcrypt registered [ 910.622392] Key type .llcrypt registered [ 910.736221] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 912.723367] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 912.785876] alg: No test for adler32 (adler32-zlib) [ 914.228427] Lustre: Lustre: Build Version: 2.17.56_50_g82e980f [ 914.843345] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [ 916.760342] Key type lgssc registered [ 918.952665] Lustre: Echo OBD driver; http://www.lustre.org/ [ 929.339403] loop: module loaded [ 929.453274] loop0: detected capacity change from 0 to 204800000 [ 929.758063] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 929.778088] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 930.741482] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 940.083061] loop0: detected capacity change from 0 to 204800000 [ 940.526973] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 940.549260] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 940.569567] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 948.230257] loop0: detected capacity change from 0 to 204800000 [ 955.664896] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 955.712680] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 957.109028] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 957.190405] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 957.363946] Lustre: lustre-MDT0000: new disk, initializing [ 957.493816] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 957.529463] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 965.635624] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 976.979970] loop1: detected capacity change from 0 to 204800000 [ 984.762294] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 985.401856] Lustre: lustre-OST07c6: new disk, initializing [ 985.409847] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 985.427526] Lustre: Skipped 1 previous similar message [ 985.438984] Lustre: 26252:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 985.577301] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 990.923566] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 990.941784] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 990.992948] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 993.516383] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 999.781043] Lustre: 25238:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 999.799262] Lustre: 25238:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 999.810562] Lustre: 25238:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 999.819581] Lustre: 25238:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 999.831178] Lustre: 25238:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 999.836408] Lustre: 25238:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 1015.792293] Lustre: 26858:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 1015.805888] Lustre: 26858:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 1015.815945] Lustre: 26858:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 1015.823143] Lustre: 26858:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 1015.837796] Lustre: 26858:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 1015.855815] Lustre: 26858:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1022.044838] Lustre: 26858:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 1022.059158] Lustre: 26858:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 1022.081216] Lustre: 26858:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 1022.095328] Lustre: 26858:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 1022.110489] Lustre: 26858:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 1022.122016] Lustre: 26858:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 1022.134424] Lustre: 26858:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 1022.147616] Lustre: 26858:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 1022.158447] Lustre: 26858:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 1022.169376] Lustre: 26858:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 1022.186377] Lustre: 26858:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1022.203652] Lustre: 26858:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 1025.462483] Lustre: 25238:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 1025.475713] Lustre: 25238:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 1025.483693] Lustre: 25238:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 1025.491405] Lustre: 25238:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 1025.500157] Lustre: 25238:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 1025.511288] Lustre: 25238:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 1025.522290] Lustre: 25238:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 1025.544327] Lustre: 25238:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 1025.568092] Lustre: 25238:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 1025.589749] Lustre: 25238:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 1025.613217] Lustre: 25238:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1025.637208] Lustre: 25238:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 1034.224891] Lustre: 25240:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 1034.242020] Lustre: 25240:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 1034.247752] Lustre: 25240:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 1034.254035] Lustre: 25240:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 1034.260434] Lustre: 25240:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 1034.267085] Lustre: 25240:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 1034.273334] Lustre: 25240:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 1034.279937] Lustre: 25240:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 1034.285557] Lustre: 25240:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 1034.292544] Lustre: 25240:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 1034.297304] Lustre: 25240:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1034.300557] Lustre: 25240:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 1043.076045] Lustre: 26858:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 1043.101233] Lustre: 26858:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 1043.112819] Lustre: 26858:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 1043.122175] Lustre: 26858:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 1043.131436] Lustre: 26858:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 1043.142533] Lustre: 26858:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 1043.151514] Lustre: 26858:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 1043.164433] Lustre: 26858:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 1043.168179] Lustre: 26858:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 1043.177753] Lustre: 26858:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 1043.186277] Lustre: 26858:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1043.204880] Lustre: 26858:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 1096.680061] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1096.708393] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 1101.491745] Lustre: server umount lustre-OST07c6 complete [ 1114.963857] Lustre: server umount lustre-MDT0000 complete [ 1144.373854] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [ 1150.788937] Key type lgssc unregistered [ 1151.701655] LNet: 28814:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1151.715767] LNetError: 28814:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1151.771167] LNet: Removed LNI 192.168.203.120@tcp [ 1154.125281] Key type .llcrypt unregistered [ 1154.138237] Key type ._llcrypt unregistered [ 1180.726423] Key type ._llcrypt registered [ 1180.728206] Key type .llcrypt registered [ 1180.974893] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 1182.957015] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1183.033222] alg: No test for adler32 (adler32-zlib) [ 1184.320579] Lustre: Lustre: Build Version: 2.17.56_50_g82e980f [ 1184.877919] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [ 1186.679164] Key type lgssc registered [ 1188.770830] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1200.684183] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1200.707074] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1202.338815] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1209.085311] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 1211.872770] LustreError: 30102:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1211.906726] LustreError: 30102:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1216.992358] LustreError: 30103:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1221.699923] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1222.113709] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1222.230544] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1228.865051] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 1239.076724] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1246.926584] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1255.728125] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1256.287735] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1265.993815] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 1272.691311] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 1278.122990] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1297.390592] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1297.419523] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1297.444465] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1298.405985] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1298.422146] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1298.449126] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1298.459653] Lustre: Skipped 1 previous similar message [ 1302.889802] Lustre: server umount lustre-OST0000 complete [ 1309.152534] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1309.187254] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1309.210361] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1314.274118] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1314.278114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1315.782021] Lustre: server umount lustre-MDT0000 complete [ 1319.394994] LustreError: 31021:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1319.425755] LustreError: 31021:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1321.031333] LustreError: 30082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785935021 with bad export cookie 14465534670003832893 [ 1321.049154] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1321.972737] Lustre: server umount lustre-MDT0001 complete [ 1341.042481] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [ 1345.239514] Key type lgssc unregistered [ 1345.663732] LNet: 33865:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1345.666938] LNetError: 33865:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1346.730905] LNet: Removed LNI 192.168.203.120@tcp [ 1348.015217] Key type .llcrypt unregistered [ 1348.017593] Key type ._llcrypt unregistered [ 1376.519123] Key type ._llcrypt registered [ 1376.526061] Key type .llcrypt registered [ 1376.645384] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 1378.482630] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1378.724386] alg: No test for adler32 (adler32-zlib) [ 1380.395103] Lustre: Lustre: Build Version: 2.17.56_50_g82e980f [ 1381.019780] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [ 1382.920875] Key type lgssc registered [ 1385.245845] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1397.873785] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1397.905371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1399.393992] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1406.221757] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 1408.486491] LustreError: 35138:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1408.518331] LustreError: 35138:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1413.608122] LustreError: 35139:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1417.615549] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1418.294188] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1423.941186] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 1433.549292] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1441.883262] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1452.346991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1452.904781] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1462.522733] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 1469.316727] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1471.464569] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1488.880706] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1488.902583] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1488.933597] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1489.897832] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1489.906957] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1489.926739] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1490.628573] Lustre: server umount lustre-OST0000 complete [ 1500.135202] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1500.140971] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1500.145529] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1500.145535] Lustre: Skipped 1 previous similar message [ 1500.156542] Lustre: Skipped 1 previous similar message [ 1505.116929] Lustre: server umount lustre-MDT0000 complete [ 1505.254053] LustreError: 35139:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1505.293486] LustreError: 35139:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1510.290622] LustreError: 37641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785935210 with bad export cookie 10023668565846755817 [ 1510.295019] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1510.305972] LustreError: 37641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1511.055982] Lustre: server umount lustre-MDT0001 complete [ 1530.068650] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [ 1533.596845] Key type lgssc unregistered [ 1533.992193] LNet: 38719:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1534.005780] LNetError: 38719:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1535.082054] LNet: Removed LNI 192.168.203.120@tcp [ 1536.729048] Key type .llcrypt unregistered [ 1536.734406] Key type ._llcrypt unregistered [ 1570.537192] Key type ._llcrypt registered [ 1570.542342] Key type .llcrypt registered [ 1570.748668] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 1572.485573] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1572.730065] alg: No test for adler32 (adler32-zlib) [ 1574.075335] Lustre: Lustre: Build Version: 2.17.56_50_g82e980f [ 1574.564344] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [ 1576.327535] Key type lgssc registered [ 1579.251279] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1590.443668] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1590.479315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1591.975132] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1597.203588] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 1601.509535] LustreError: 40138:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1601.539100] LustreError: 40138:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1606.630938] LustreError: 40139:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1610.370445] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1611.109555] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1618.049497] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 1628.960974] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1637.127450] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1645.727857] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1646.331331] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1655.127541] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 1661.768092] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1662.779168] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1672.168991] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1672.172949] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1672.205071] Lustre: Skipped 2 previous similar messages [ 1672.232975] Lustre: Skipped 2 previous similar messages [ 1675.181317] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.20@tcp (stopping) [ 1677.287662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1677.297098] Lustre: Skipped 2 previous similar messages [ 1678.540864] Lustre: server umount lustre-MDT0000 complete [ 1680.264514] LustreError: 40131:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1682.407628] LustreError: 40131:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1682.430526] LustreError: 40131:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1683.517882] LustreError: 40117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785935384 with bad export cookie 5682567251038788688 [ 1683.527502] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1683.546404] LustreError: 40117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1685.420294] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.20@tcp (stopping) [ 1687.522698] LustreError: 41200:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1687.525857] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1687.531467] LustreError: 41200:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1690.240564] Lustre: server umount lustre-MDT0001 complete [ 1707.316043] Lustre: server umount lustre-OST0000 complete [ 1733.901125] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [ 1739.508428] Key type lgssc unregistered [ 1739.922463] LNet: 44031:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1739.936512] LNetError: 44031:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1739.968221] LNet: Removed LNI 192.168.203.120@tcp [ 1741.144485] Key type .llcrypt unregistered [ 1741.148358] Key type ._llcrypt unregistered [ 1774.005510] Key type ._llcrypt registered [ 1774.013831] Key type .llcrypt registered [ 1774.211597] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 1775.589259] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1775.907068] alg: No test for adler32 (adler32-zlib) [ 1777.215177] Lustre: Lustre: Build Version: 2.17.56_50_g82e980f [ 1777.462722] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [ 1779.138353] Key type lgssc registered [ 1781.022191] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1791.576094] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1791.627557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1793.458986] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1801.298696] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 1802.721608] LustreError: 45453:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1802.749375] LustreError: 45453:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1807.854561] LustreError: 45452:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1812.963971] LustreError: 45453:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1813.022420] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1813.511298] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1820.019791] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 1830.383510] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1838.349895] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1848.354686] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1848.906181] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1858.347631] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 1863.315768] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1869.014858] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1877.411810] Lustre: lustre-OST0000: Not available for connect from 192.168.203.20@tcp (stopping) [ 1879.010535] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1879.025831] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1879.040317] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1879.078404] Lustre: Skipped 1 previous similar message [ 1882.503453] Lustre: lustre-OST0000: Not available for connect from 192.168.203.20@tcp (stopping) [ 1882.512048] Lustre: Skipped 1 previous similar message [ 1883.600938] Lustre: server umount lustre-OST0000 complete [ 1905.632518] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1905.649597] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1905.685932] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1908.705432] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1912.473422] Lustre: server umount lustre-MDT0000 complete [ 1913.837047] LustreError: 45452:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1913.866701] LustreError: 45452:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1917.697421] LustreError: 45431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785935618 with bad export cookie 4927338035869401379 [ 1917.712988] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1917.720019] LustreError: 45431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1932.255376] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1933.287424] Lustre: server umount lustre-MDT0001 complete [ 1951.994988] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [ 1956.727865] Key type lgssc unregistered [ 1957.295504] LNet: 49192:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1957.302429] LNetError: 49192:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1957.352710] LNet: Removed LNI 192.168.203.120@tcp [ 1958.731888] Key type .llcrypt unregistered [ 1958.733500] Key type ._llcrypt unregistered [ 1991.719472] Key type ._llcrypt registered [ 1991.728468] Key type .llcrypt registered [ 1991.849414] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 1993.347709] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1993.668213] alg: No test for adler32 (adler32-zlib) [ 1995.018807] Lustre: Lustre: Build Version: 2.17.56_50_g82e980f [ 1995.442118] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [ 1997.239168] Key type lgssc registered [ 1999.783229] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2009.725846] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2009.758725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2011.554754] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2018.294579] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 2021.857912] LustreError: 50611:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2021.884591] LustreError: 50611:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2026.978250] LustreError: 50612:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2030.852850] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2031.545232] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2038.173511] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 2049.501937] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2060.484827] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2070.673277] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2071.264724] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2079.779977] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 2081.545542] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 2089.994199] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2098.044667] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 2103.312197] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 2117.193937] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-05 9h16m54s [ 2119.492412] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 2122.721775] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2122.733823] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2122.746834] Lustre: Skipped 1 previous similar message [ 2122.762466] Lustre: Skipped 2 previous similar messages [ 2123.709435] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.20@tcp (stopping) [ 2124.792708] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.20@tcp (stopping) [ 2124.808335] Lustre: Skipped 3 previous similar messages [ 2129.109188] Lustre: server umount lustre-MDT0000 complete [ 2130.402867] LustreError: 50611:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2130.439709] LustreError: 50611:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2134.154225] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 2136.565810] LustreError: 52145:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2136.589771] LustreError: 52145:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2142.698038] LustreError: 50611:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2142.731839] LustreError: 50611:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2143.248102] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2143.526666] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2144.061588] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2144.142241] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 2145.093304] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2151.010285] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 2155.770616] Lustre: DEBUG MARKER: Wait for df (36804) ... [ 2157.807614] Lustre: DEBUG MARKER: done [ 2164.705266] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2164.705662] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2164.719768] LustreError: Skipped 1 previous similar message [ 2164.720604] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2164.746352] Lustre: Skipped 2 previous similar messages [ 2170.501619] Lustre: server umount lustre-OST0000 complete [ 2175.969921] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2175.973949] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2175.990908] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2175.990915] Lustre: Skipped 3 previous similar messages [ 2176.035301] Lustre: Skipped 1 previous similar message [ 2181.660563] Lustre: server umount lustre-MDT0000 complete [ 2183.652682] LustreError: 50608:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2183.674268] LustreError: 50608:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2186.532471] LustreError: 50591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785935887 with bad export cookie 6945149905605783126 [ 2186.545481] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2186.553056] LustreError: 50591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2186.984619] Lustre: server umount lustre-MDT0001 complete [ 2205.724603] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [ 2208.465291] Key type lgssc unregistered [ 2208.790461] LNet: 55555:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2208.810694] LNetError: 55555:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2208.854330] LNet: Removed LNI 192.168.203.120@tcp [ 2211.079734] Key type .llcrypt unregistered [ 2211.086683] Key type ._llcrypt unregistered [ 2273.502432] Key type ._llcrypt registered [ 2273.504100] Key type .llcrypt registered [ 2273.640200] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 2275.717869] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2275.752347] alg: No test for adler32 (adler32-zlib) [ 2277.035742] Lustre: Lustre: Build Version: 2.17.56_50_g82e980f [ 2277.383942] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [ 2279.144499] Key type lgssc registered [ 2281.102523] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2292.199540] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2292.253748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2293.779265] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2293.838479] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2294.340404] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2305.471676] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 2317.558609] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2317.698094] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2317.741983] Lustre: 57768:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2318.172351] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2325.794784] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 2335.557414] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2342.568250] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2351.162549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2351.428167] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2351.843152] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2359.086141] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 2360.504750] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 2371.202478] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2378.686594] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-05 9h21m17s [ 2380.621237] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 2389.673215] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 2392.796745] Lustre: DEBUG MARKER: Creating a test file and stat it [ 2392.871140] LustreError: 57776:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 2413.051096] Lustre: lustre-MDT0000: Client 64afec6e-20ab-4141-af59-a6e3ff2903aa (at 192.168.203.20@tcp) reconnecting [ 2432.911908] LustreError: 57776:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 2435.505550] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 2439.162220] Lustre: DEBUG MARKER: done [ 2451.432575] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2451.434852] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2451.435372] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2451.460107] Lustre: Skipped 1 previous similar message [ 2451.490344] LustreError: Skipped 1 previous similar message [ 2454.416476] Lustre: server umount lustre-OST0000 complete [ 2461.664528] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2461.668969] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2461.675908] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2461.675916] Lustre: Skipped 1 previous similar message [ 2462.693645] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2462.699994] Lustre: Skipped 2 previous similar messages [ 2466.187698] Lustre: server umount lustre-MDT0000 complete [ 2471.954675] LustreError: 60531:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785936172 with bad export cookie 6461892952887996967 [ 2471.968076] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2471.978829] LustreError: 60531:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2473.350866] Lustre: server umount lustre-MDT0001 complete [ 2492.449590] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [ 2498.690727] Key type lgssc unregistered [ 2499.406980] LNet: 62141:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2499.415364] LNetError: 62141:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2499.455377] LNet: Removed LNI 192.168.203.120@tcp [ 2501.675262] Key type .llcrypt unregistered [ 2501.679300] Key type ._llcrypt unregistered [ 2574.898868] Key type ._llcrypt registered [ 2574.903702] Key type .llcrypt registered [ 2575.080254] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 2576.981660] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2577.106273] alg: No test for adler32 (adler32-zlib) [ 2578.314199] Lustre: Lustre: Build Version: 2.17.56_50_g82e980f [ 2578.642175] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [ 2580.407141] Key type lgssc registered [ 2582.047777] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2588.815220] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2588.872675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2590.511712] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2590.612692] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2591.043604] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2593.078521] Lustre: Failing over lustre-MDT0000 [ 2594.169688] Lustre: server umount lustre-MDT0000 complete [ 2608.817564] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 09:25:06 (1785936306) [ 2618.822674] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2619.836642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2627.070601] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 2641.079489] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2641.217800] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2641.293885] Lustre: 64948:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2641.658644] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2649.397046] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 2660.907148] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2669.051974] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2678.879159] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2679.220235] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2679.633109] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2687.181248] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2687.710223] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 2698.664970] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2705.164281] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2713.571755] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2713.589801] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2713.619149] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2718.177295] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2718.180623] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2718.187189] Lustre: Skipped 1 previous similar message [ 2718.212313] Lustre: Skipped 1 previous similar message [ 2720.057069] Lustre: server umount lustre-MDT0000 complete [ 2725.532369] LustreError: 64941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785936426 with bad export cookie 3087407917115682245 [ 2725.552335] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2725.552872] LustreError: 64941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2726.133864] Lustre: server umount lustre-MDT0001 complete [ 2730.085757] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2735.597242] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2744.800176] Lustre: 63868:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785936429/real 1785936429] req@000000003b78cc2b x1872689919735552/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785936445 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2744.850504] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2746.592850] Lustre: 63869:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785936431/real 1785936431] req@00000000232b2909 x1872689919735680/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785936447 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2748.231573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2749.920683] Lustre: 63866:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785936434/real 1785936434] req@000000006e1b6ce5 x1872689919735808/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785936450 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2750.624052] LustreError: 68708:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 2750.649099] LustreError: 68708:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@0000000057daf0ac x1872689919735936/t0(0) o250->MGC192.168.203.120@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1785936451 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2750.691077] LustreError: 68708:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 2751.645102] LustreError: 66797:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2751.672892] LustreError: 66797:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2751.807785] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2752.035173] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2752.735511] Lustre: 63867:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785936436/real 1785936436] req@00000000f6e62e05 x1872689919736064/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785936452 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2753.826799] LustreError: 66798:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2754.872344] LustreError: 66797:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2754.935727] LustreError: 66797:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2756.897058] Lustre: 63868:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785936441/real 1785936441] req@000000002595ff7d x1872689919736192/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785936457 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2759.654246] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 2760.176705] LustreError: 68721:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2760.220515] LustreError: 68721:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2765.299819] LustreError: 68720:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2765.333742] LustreError: 68720:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2773.485825] LustreError: 66798:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2773.519646] LustreError: 66798:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2773.539181] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2773.800697] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2774.387790] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2779.642575] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2782.580879] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 2795.775742] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2805.387373] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2815.465983] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2815.476675] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2815.479973] Lustre: Skipped 1 previous similar message [ 2815.493539] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2815.514761] Lustre: Skipped 3 previous similar messages [ 2819.531611] Lustre: server umount lustre-MDT0000 complete [ 2820.580843] LustreError: 66798:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2820.617686] LustreError: 66798:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2824.292898] LustreError: 67864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785936524 with bad export cookie 3087407917115684520 [ 2824.311408] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2824.312582] LustreError: 67864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2824.865839] Lustre: server umount lustre-MDT0001 complete [ 2835.562576] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2842.079267] Lustre: 63867:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785936526/real 1785936526] req@00000000da33cdd8 x1872689919768192/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785936542 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2842.118282] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2842.140092] Lustre: Skipped 1 previous similar message [ 2850.743354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2868.770352] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x2ad8ad249068bc0b [ 2868.789504] Lustre: MGC192.168.203.120@tcp: Connection restored to 0@lo (at 0@lo) [ 2869.282747] LustreError: 66798:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2869.399644] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2869.491862] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2871.403244] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2875.196916] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 2887.055510] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2892.773076] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2893.777246] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 2904.735782] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2913.978493] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2928.626264] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2928.633313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2928.647644] Lustre: Skipped 1 previous similar message [ 2928.660705] Lustre: Skipped 1 previous similar message [ 2928.661015] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2930.775168] Lustre: server umount lustre-MDT0000 complete [ 2933.730450] LustreError: 72355:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2933.760338] LustreError: 72355:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 2936.367275] LustreError: 65692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785936636 with bad export cookie 3087407917115685899 [ 2936.368522] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2936.378604] LustreError: 65692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2937.002984] Lustre: server umount lustre-MDT0001 complete [ 2946.552681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2952.524153] Lustre: DEBUG MARKER: files compared the same [ 2965.716939] Lustre: server umount lustre-OST0000 complete [ 2993.911247] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [ 2999.612492] Key type lgssc unregistered [ 3000.275158] LNet: 74718:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3000.290665] LNetError: 74718:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3000.334526] LNet: Removed LNI 192.168.203.120@tcp [ 3002.291209] Key type .llcrypt unregistered [ 3002.303654] Key type ._llcrypt unregistered [ 3038.935429] Key type ._llcrypt registered [ 3038.938348] Key type .llcrypt registered [ 3039.176678] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 3040.739041] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3040.784295] alg: No test for adler32 (adler32-zlib) [ 3042.557142] Lustre: Lustre: Build Version: 2.17.56_50_g82e980f [ 3043.024263] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [ 3044.831179] Key type lgssc registered [ 3046.529594] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3057.178261] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3057.194616] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3058.728985] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3066.136453] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug +malloc all [ 3068.900722] LustreError: 76137:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3068.943943] LustreError: 76137:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3074.017289] LustreError: 76138:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3077.483545] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3077.969245] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3084.282206] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug +malloc all [ 3095.419139] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3104.048250] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3113.604435] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3114.036930] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3123.813801] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug +malloc all [ 3129.387702] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 3133.916372] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3144.672188] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3144.705133] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3144.716573] Lustre: Skipped 1 previous similar message [ 3149.793964] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3150.448824] Lustre: server umount lustre-OST0000 complete [ 3160.047727] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3160.061641] Lustre: Skipped 1 previous similar message [ 3160.067630] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3160.071927] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3160.078782] Lustre: Skipped 2 previous similar messages [ 3162.634134] Lustre: server umount lustre-MDT0000 complete [ 3165.151986] LustreError: 76137:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3165.168629] LustreError: 76137:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3166.421681] LustreError: 76119:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785936867 with bad export cookie 11259131685269266348 [ 3166.425287] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3166.433491] LustreError: 76119:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3167.058650] Lustre: server umount lustre-MDT0001 complete [ 3183.740514] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [ 3187.197407] Key type lgssc unregistered [ 3187.450712] LNet: 79679:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3187.460594] LNetError: 79679:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3187.500960] LNet: Removed LNI 192.168.203.120@tcp [ 3188.856480] Key type .llcrypt unregistered [ 3188.864041] Key type ._llcrypt unregistered [ 3224.444845] Key type ._llcrypt registered [ 3224.449833] Key type .llcrypt registered [ 3224.629439] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 3226.218840] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3226.369094] alg: No test for adler32 (adler32-zlib) [ 3227.802787] Lustre: Lustre: Build Version: 2.17.56_50_g82e980f [ 3228.348950] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [ 3230.183160] Key type lgssc registered [ 3232.577647] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3245.405676] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3245.435370] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3261.151387] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3266.271280] LustreError: 81072:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.120@tcp: failed processing log, type 4: rc = -110 [ 3293.023459] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3302.680675] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 3315.081890] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3327.611286] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3327.929363] Lustre: *** cfs_fail_loc=706, val=0*** [ 3328.886617] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3329.021516] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 3334.117599] LustreError: 81097:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3334.163427] LustreError: 81097:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3335.061836] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 3339.232441] LustreError: 81098:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3344.359145] LustreError: 81097:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3346.400312] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3346.859819] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3353.215887] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 3362.946496] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3370.920660] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3375.072495] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3375.086780] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3375.107432] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3377.636554] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3377.713279] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3377.748498] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3380.083268] Lustre: server umount lustre-OST0000 complete [ 3387.872078] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3387.887573] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3387.939118] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3387.948216] Lustre: Skipped 2 previous similar messages [ 3392.907647] Lustre: server umount lustre-MDT0000 complete [ 3394.017140] LustreError: 82166:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3394.066784] LustreError: 82166:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3398.914147] LustreError: 81079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785937099 with bad export cookie 16327313400391276940 [ 3398.917084] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3398.928569] LustreError: 81079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3399.764989] Lustre: server umount lustre-MDT0001 complete [ 3416.317758] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [ 3420.996370] Key type lgssc unregistered [ 3421.603737] LNet: 84697:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3421.617426] LNetError: 84697:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3421.681761] LNet: Removed LNI 192.168.203.120@tcp [ 3423.521965] Key type .llcrypt unregistered [ 3423.527293] Key type ._llcrypt unregistered [ 3456.684428] Key type ._llcrypt registered [ 3456.699402] Key type .llcrypt registered [ 3456.896102] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 3458.285586] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3458.422205] alg: No test for adler32 (adler32-zlib) [ 3459.740623] Lustre: Lustre: Build Version: 2.17.56_50_g82e980f [ 3460.046430] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [ 3461.839161] Key type lgssc registered [ 3463.671553] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3472.588980] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3472.643860] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3480.203629] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 3492.221746] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3493.635356] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3499.101269] LustreError: 86807:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3499.125156] LustreError: 86807:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3501.425712] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 3504.114569] LustreError: 86810:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3504.204588] LustreError: 86810:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3509.216645] LustreError: 86811:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3509.253591] LustreError: 86811:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3514.336845] LustreError: 86805:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3514.369641] LustreError: 86805:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3516.637973] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3517.431506] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3518.452059] LustreError: 86806:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3518.481328] LustreError: 86806:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 3527.081299] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 3528.674365] LustreError: 86811:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3528.697836] LustreError: 86811:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 3540.447154] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3540.456415] Lustre: MGS: Client 90d637b6-d150-46d5-b6e3-022b10472d95 (at 0@lo) reconnecting [ 3540.473350] LustreError: 86821:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 3540.897782] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3541.103020] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 3550.769961] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 3571.684719] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3571.711372] LustreError: 87534:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3571.733450] LustreError: 87534:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 3571.867279] Lustre: server umount lustre-OST0000 complete [ 3581.919974] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3581.924225] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3581.930733] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3581.939966] Lustre: Skipped 2 previous similar messages [ 3581.956758] Lustre: Skipped 1 previous similar message [ 3583.978640] Lustre: server umount lustre-MDT0000 complete [ 3588.861328] Lustre: server umount lustre-MDT0001 complete [ 3593.870580] Lustre: server umount MGS complete [ 3610.710037] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 09:41:49 (1785937309) [ 3650.599694] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [ 3663.466798] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 3710.839441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3717.281755] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 3731.131347] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3731.376227] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 3731.403153] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3731.475956] Lustre: lustre-MDT0001: new disk, initializing [ 3731.781160] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3731.840984] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3731.850478] Lustre: 94004:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 3731.861859] LustreError: 94004:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 3738.898488] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 3754.917406] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3755.489459] Lustre: lustre-OST0000: new disk, initializing [ 3755.497660] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3755.520061] Lustre: 94772:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3755.634242] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3764.835805] Lustre: 95071:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000aa22f3b6 x1872690843531392/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1785937481 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3765.020623] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 3778.743522] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3785.184134] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3788.365634] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3788.380956] Lustre: MGS: Client 46de0b1a-3190-4ef4-a35f-5f3916539248 (at 0@lo) reconnecting [ 3788.450160] Lustre: 93271:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3788.471195] LustreError: 93990:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 3788.538289] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3788.720362] Lustre: lustre-MDT0000: new disk, initializing [ 3789.106559] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3789.161747] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3790.246440] Lustre: 95845:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@000000008b3152a1 x1872690843539328/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1785937506 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3795.381358] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3795.402434] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3795.636954] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 3797.599794] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 3826.657102] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3826.667324] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3826.692484] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3830.247662] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3830.249047] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3833.000198] Lustre: server umount lustre-OST0000 complete [ 3840.485203] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3840.486462] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3840.500165] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3840.511752] Lustre: Skipped 1 previous similar message [ 3840.523966] Lustre: Skipped 1 previous similar message [ 3845.601756] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3845.605179] Lustre: Skipped 1 previous similar message [ 3845.661505] Lustre: server umount lustre-MDT0000 complete [ 3849.700541] LustreError: 93976:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3849.727858] LustreError: 93976:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 3859.156702] Lustre: server umount lustre-MDT0001 complete [ 3864.662685] Lustre: server umount MGS complete [ 3881.347195] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 09:46:19 (1785937579) [ 3905.833866] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [ 3909.553586] Key type lgssc unregistered [ 3909.933958] LNet: 98349:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3909.942308] LNetError: 98349:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3909.982131] LNet: Removed LNI 192.168.203.120@tcp [ 3911.026793] Key type .llcrypt unregistered [ 3911.029978] Key type ._llcrypt unregistered [ 3933.485537] Key type ._llcrypt registered [ 3933.496323] Key type .llcrypt registered [ 3933.681921] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 3936.481978] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3936.534285] alg: No test for adler32 (adler32-zlib) [ 3937.787129] Lustre: Lustre: Build Version: 2.17.56_50_g82e980f [ 3938.129338] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [ 3939.906389] Key type lgssc registered [ 3941.322633] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3951.832434] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3951.861086] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3953.553607] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3959.641112] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 3963.874384] LustreError: 99575:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3963.891142] LustreError: 99575:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3969.006730] LustreError: 99574:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3972.668975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3972.987077] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3978.682761] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 3989.405447] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3997.128309] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4005.863484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4006.316283] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4013.194709] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 4021.657649] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4023.737703] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 4038.113137] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4038.119630] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4038.133698] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4039.137920] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4039.140046] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4039.176532] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4044.019873] Lustre: server umount lustre-OST0000 complete [ 4049.379545] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4049.390506] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4049.409953] Lustre: Skipped 1 previous similar message [ 4049.426255] Lustre: Skipped 2 previous similar messages [ 4054.506075] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4054.523086] Lustre: Skipped 1 previous similar message [ 4055.135076] Lustre: server umount lustre-MDT0000 complete [ 4059.484882] LustreError: 99556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785937760 with bad export cookie 649723716339079463 [ 4059.494903] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4059.499246] LustreError: 99556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4060.305882] Lustre: server umount lustre-MDT0001 complete [ 4075.087872] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [ 4078.997308] Key type lgssc unregistered [ 4079.408674] LNet: 103132:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4079.431453] LNetError: 103132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4079.450753] LNet: Removed LNI 192.168.203.120@tcp [ 4080.732251] Key type .llcrypt unregistered [ 4080.736054] Key type ._llcrypt unregistered [ 4101.709267] Key type ._llcrypt registered [ 4101.710612] Key type .llcrypt registered [ 4101.878688] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 4103.604431] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4103.704240] alg: No test for adler32 (adler32-zlib) [ 4104.834758] Lustre: Lustre: Build Version: 2.17.56_50_g82e980f [ 4105.151924] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [ 4106.887106] Key type lgssc registered [ 4108.699338] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4117.782492] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4117.789629] LustreError: 104212:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 4117.830924] LustreError: 104219:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 4117.842893] LustreError: 104212:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4958 [ 4117.849847] LustreError: 104212:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 4117.861656] LustreError: 104212:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 4117.866792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4119.680026] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4129.967177] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4130.272509] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 4130.471121] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4136.771348] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 4145.055844] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing lsmod [ 4149.965849] LustreError: 105419:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 4150.019687] LustreError: 105426:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 4150.029303] LustreError: 105419:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4960 [ 4150.041904] LustreError: 105419:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 4150.064112] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4150.066157] LustreError: 105419:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 4150.711727] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4155.682033] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4155.684430] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4155.710956] Lustre: Skipped 1 previous similar message [ 4155.732480] Lustre: Skipped 2 previous similar messages [ 4160.996355] LustreError: 104257:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4161.024838] LustreError: 104257:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 4161.084967] Lustre: server umount lustre-MDT0000 complete [ 4166.119178] LustreError: 104257:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4166.150024] LustreError: 104257:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4167.002140] LustreError: 104231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785937867 with bad export cookie 5226945055856043541 [ 4167.014118] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4167.019959] LustreError: 104231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4168.095245] LustreError: 104688:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000348c3931 x1872691519837440/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4168.898987] Lustre: server umount lustre-MDT0001 complete [ 4184.346730] Lustre: server umount lustre-OST0000 complete [ 4194.806389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4195.412380] LustreError: 106493:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4195.471076] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4200.929553] LustreError: 106494:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4202.237309] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 4206.048493] LustreError: 106493:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4212.915487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4213.430090] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4218.953353] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 4230.528837] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4238.049482] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4248.883594] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4249.614171] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4260.074961] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 4265.995747] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 4271.140537] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4281.316675] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4281.318593] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4281.322540] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4281.340367] Lustre: Skipped 2 previous similar messages [ 4286.348384] Lustre: server umount lustre-OST0000 complete [ 4291.553972] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4291.577870] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4291.589161] Lustre: Skipped 2 previous similar messages [ 4296.672959] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4296.687928] Lustre: Skipped 1 previous similar message [ 4305.375320] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4305.909416] Lustre: server umount lustre-MDT0000 complete [ 4306.930391] LustreError: 106494:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4306.951908] LustreError: 106494:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4310.843503] LustreError: 106474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785938011 with bad export cookie 5226945055856044241 [ 4310.858438] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4311.464622] Lustre: server umount lustre-MDT0001 complete [ 4330.495264] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [ 4334.513523] Key type lgssc unregistered [ 4334.858324] LNet: 110053:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4334.865155] LNetError: 110053:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4334.889671] LNet: Removed LNI 192.168.203.120@tcp [ 4335.897304] Key type .llcrypt unregistered [ 4335.903329] Key type ._llcrypt unregistered [ 4369.716327] Key type ._llcrypt registered [ 4369.719332] Key type .llcrypt registered [ 4369.876390] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 4372.735815] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4372.814178] alg: No test for adler32 (adler32-zlib) [ 4374.251137] Lustre: Lustre: Build Version: 2.17.56_50_g82e980f [ 4374.618730] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [ 4376.455164] Key type lgssc registered [ 4379.268870] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4397.453781] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4397.499411] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4399.686553] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4406.555197] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 4409.834873] LustreError: 111476:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4409.861933] LustreError: 111476:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4414.949119] LustreError: 111477:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4419.652904] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4420.085457] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 4420.246894] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4425.956978] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 4437.380267] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4448.939149] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4460.780671] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4461.367779] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4469.599527] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 4470.037099] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 4479.834330] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4486.111771] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 4492.308996] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 4500.453227] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4500.470265] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4500.478060] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4500.494877] Lustre: Skipped 1 previous similar message [ 4503.751564] Lustre: server umount lustre-OST0000 complete [ 4510.699554] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4510.726490] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4510.747846] Lustre: Skipped 2 previous similar messages [ 4515.309482] Lustre: server umount lustre-MDT0000 complete [ 4515.817572] LustreError: 111477:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4515.845229] LustreError: 111477:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4520.234932] LustreError: 111456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785938220 with bad export cookie 2705331028305588762 [ 4520.258141] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4520.265435] LustreError: 111456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4520.735995] Lustre: server umount lustre-MDT0001 complete [ 4540.085372] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [ 4544.577201] Key type lgssc unregistered [ 4545.063974] LNet: 115147:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4545.072264] LNetError: 115147:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4545.094898] LNet: Removed LNI 192.168.203.120@tcp [ 4546.309643] Key type .llcrypt unregistered [ 4546.314622] Key type ._llcrypt unregistered [ 4572.356722] Key type ._llcrypt registered [ 4572.373119] Key type .llcrypt registered [ 4572.663831] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 4574.850782] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4575.075866] alg: No test for adler32 (adler32-zlib) [ 4577.307956] Lustre: Lustre: Build Version: 2.17.56_50_g82e980f [ 4577.665007] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [ 4579.400863] Key type lgssc registered [ 4580.940386] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4594.328499] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4594.342555] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4596.028125] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4605.286398] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 4605.423231] LustreError: 116370:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4605.445843] LustreError: 116370:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4610.544675] LustreError: 116370:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4615.674555] LustreError: 116369:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4616.982326] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4617.364767] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4624.551970] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 4634.001410] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4643.192456] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4652.167696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4652.902305] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4662.293244] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 4666.166878] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 4671.934453] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4682.721320] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4682.730403] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4682.743825] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4686.830882] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4686.836643] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4686.844111] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4689.019497] Lustre: server umount lustre-OST0000 complete [ 4697.056434] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4697.083366] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4697.091446] Lustre: Skipped 2 previous similar messages [ 4701.381880] Lustre: server umount lustre-MDT0000 complete [ 4702.176383] LustreError: 116370:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4702.210439] LustreError: 116370:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4707.301166] LustreError: 118200:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4707.339474] LustreError: 118200:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4708.534920] LustreError: 116351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785938409 with bad export cookie 17829505173621825696 [ 4708.548496] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4708.553486] LustreError: 116351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4709.697571] Lustre: server umount lustre-MDT0001 complete [ 4732.562206] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [ 4736.805535] Key type lgssc unregistered [ 4737.159623] LNet: 119933:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4737.163818] LNetError: 119933:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4737.190231] LNet: Removed LNI 192.168.203.120@tcp [ 4738.195875] Key type .llcrypt unregistered [ 4738.199922] Key type ._llcrypt unregistered [ 4771.124973] Key type ._llcrypt registered [ 4771.128163] Key type .llcrypt registered [ 4771.316521] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 4772.890964] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4773.156567] alg: No test for adler32 (adler32-zlib) [ 4774.545590] Lustre: Lustre: Build Version: 2.17.56_50_g82e980f [ 4775.007402] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [ 4776.847217] Key type lgssc registered [ 4778.875393] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4790.894423] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4790.922627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4792.406461] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4797.409852] LustreError: 121364:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4797.430756] LustreError: 121364:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4797.488935] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 4801.510882] LustreError: 121365:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4806.633107] LustreError: 121364:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4808.582508] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4809.069619] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4815.304846] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 4825.141927] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4832.848939] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4843.821891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4844.921900] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4854.944492] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 4862.258866] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 4865.286128] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4875.434754] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 4881.914042] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 4881.927335] Lustre: Skipped 2 previous similar messages [ 4884.603823] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 4884.637050] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 4888.177133] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 4888.194274] Lustre: Skipped 2 previous similar messages [ 4892.200241] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4892.203670] Lustre: Skipped 1 previous similar message [ 4899.248119] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.20@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 4899.259783] Lustre: Skipped 2 previous similar messages [ 4900.609516] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.20@tcp 192.168.0.[2,10]@tcp [ 4912.730277] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.20/24@tcp in log lustre-MDT0000 [ 4912.738481] Lustre: Skipped 2 previous similar messages [ 4918.937145] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.20/24@tcp [ 4918.943727] Lustre: Skipped 1 previous similar message [ 4932.069974] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4932.089289] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4932.132200] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4934.112837] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4934.114417] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4934.131520] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4934.162706] Lustre: Skipped 1 previous similar message [ 4936.860808] Lustre: server umount lustre-OST0000 complete [ 4942.317241] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4942.324845] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4942.344897] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4944.352648] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4944.359987] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4944.389607] Lustre: Skipped 1 previous similar message [ 4948.223846] Lustre: server umount lustre-MDT0000 complete [ 4949.489492] LustreError: 122430:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4949.507802] LustreError: 122430:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4952.851923] LustreError: 121346:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785938653 with bad export cookie 12566247204844323795 [ 4952.861577] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4952.868602] LustreError: 121346:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4953.592740] Lustre: server umount lustre-MDT0001 complete [ 4966.970168] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [ 4970.529335] Key type lgssc unregistered [ 4970.954919] LNet: 126123:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4970.961846] LNetError: 126123:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4970.986688] LNet: Removed LNI 192.168.203.120@tcp [ 4972.082798] Key type .llcrypt unregistered [ 4972.087081] Key type ._llcrypt unregistered [ 5007.643402] Key type ._llcrypt registered [ 5007.648187] Key type .llcrypt registered [ 5007.778739] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 5009.266739] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5009.552410] alg: No test for adler32 (adler32-zlib) [ 5010.628940] Lustre: Lustre: Build Version: 2.17.56_50_g82e980f [ 5011.069878] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [ 5012.831147] Key type lgssc registered [ 5014.380888] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5023.438938] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5023.480491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5025.085991] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 5025.133378] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.20/24@tcp [ 5025.152357] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5031.342567] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 5034.465510] LustreError: 127684:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5034.497129] LustreError: 127684:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5039.583818] LustreError: 127685:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5044.706618] LustreError: 127684:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5045.949060] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5046.600554] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 5046.661877] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.20/24@tcp [ 5046.755318] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5055.702653] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 5067.880691] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5075.209422] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5083.460148] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5084.123387] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5092.197076] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 5095.420851] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 5101.048300] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5115.876214] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5115.901224] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5120.996725] LustreError: 130007:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5121.015667] LustreError: 130007:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5121.153483] Lustre: server umount lustre-OST0000 complete [ 5128.678740] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5128.688376] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5128.698606] Lustre: Skipped 1 previous similar message [ 5128.711946] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5128.719433] Lustre: Skipped 1 previous similar message [ 5131.232862] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5131.234024] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5132.868438] Lustre: server umount lustre-MDT0000 complete [ 5136.368866] LustreError: 127680:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5136.389408] LustreError: 127680:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5137.575370] LustreError: 127664:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785938838 with bad export cookie 16073153051106939590 [ 5137.592206] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5138.189639] Lustre: server umount lustre-MDT0001 complete [ 5152.410679] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [ 5156.824839] Key type lgssc unregistered [ 5157.288050] LNet: 131282:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5157.292201] LNetError: 131282:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5157.315823] LNet: Removed LNI 192.168.203.120@tcp [ 5158.835695] Key type .llcrypt unregistered [ 5158.842344] Key type ._llcrypt unregistered [ 5202.556833] Key type ._llcrypt registered [ 5202.559391] Key type .llcrypt registered [ 5202.765893] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [ 5221.336713] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 5224.029850] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5224.063035] alg: No test for adler32 (adler32-zlib) [ 5225.535936] Lustre: Lustre: Build Version: 2.17.56_50_g82e980f [ 5225.945235] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [ 5227.791655] Key type lgssc registered [ 5229.520153] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5278.115762] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5278.164030] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5279.452733] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5279.481399] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5279.608849] Lustre: lustre-MDT0000: new disk, initializing [ 5279.927690] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5279.987464] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5286.722713] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 5300.536950] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5300.639298] Lustre: 134686:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5300.677876] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5300.681225] Lustre: Skipped 1 previous similar message [ 5300.736362] Lustre: lustre-MDT0001: new disk, initializing [ 5300.810245] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5300.875743] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5300.892301] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5307.643719] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 5320.237726] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5329.594648] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5339.859962] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5340.844561] Lustre: lustre-OST0000: new disk, initializing [ 5340.861794] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5340.873598] Lustre: 136657:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5341.077018] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5348.488575] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5348.540471] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5348.757758] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5354.577187] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 5374.044110] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5384.853274] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5385.342153] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5393.000713] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5393.403510] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5400.043584] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5400.048442] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5400.055649] LustreError: Skipped 1 previous similar message [ 5400.056785] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5400.084554] Lustre: Skipped 1 previous similar message [ 5405.164582] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5405.176860] Lustre: Skipped 2 previous similar messages [ 5410.271115] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5410.277704] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5410.297040] Lustre: Skipped 1 previous similar message [ 5410.647490] Lustre: server umount lustre-OST0000 complete [ 5418.464476] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5418.472481] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5418.508765] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5420.521524] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5425.634682] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5425.637347] Lustre: Skipped 2 previous similar messages [ 5430.239201] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5430.747349] Lustre: server umount lustre-MDT0000 complete [ 5435.850814] LustreError: 134679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785939136 with bad export cookie 14202760842780894823 [ 5435.859792] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5435.869307] LustreError: 134679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5436.408685] Lustre: server umount lustre-MDT0001 complete [ 5446.784710] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5447.559239] LustreError: 138881:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5447.579672] LustreError: 138881:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5447.669441] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5452.773892] LustreError: 138882:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5453.376663] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 5457.891285] LustreError: 138881:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5464.042926] LustreError: 138881:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5464.071514] LustreError: 138881:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5465.542321] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5466.047617] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5472.144648] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 5483.003236] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5491.143198] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5500.294679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5500.481684] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5508.732491] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 5519.598683] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5527.004601] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5527.732726] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5538.040156] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5540.062897] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5556.117098] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5556.631995] LustreError: 140709:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5556.693762] Lustre: lustre-OST0001: new disk, initializing [ 5556.731655] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5556.755625] Lustre: 142258:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5557.036059] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5565.588544] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5565.628920] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5565.780156] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5567.715931] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 5579.562676] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5579.715755] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5585.822270] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5586.067870] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5594.856225] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8a02c64c0800.ost_server_uuid 50 [ 5597.397546] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8a02c64c0800.ost_server_uuid in IDLE state after 0 sec [ 5622.242725] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5622.249737] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5622.262137] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5622.262146] Lustre: Skipped 1 previous similar message [ 5622.270798] Lustre: Skipped 1 previous similar message [ 5634.527148] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5634.714431] Lustre: server umount lustre-OST0001 complete [ 5637.626701] LustreError: 140707:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5637.638389] LustreError: 140707:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5642.210261] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5642.217870] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5642.255541] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5642.262500] Lustre: Skipped 5 previous similar messages [ 5646.052561] Lustre: server umount lustre-OST0000 complete [ 5652.960350] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5652.986656] Lustre: Skipped 2 previous similar messages [ 5666.271355] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5666.855855] Lustre: server umount lustre-MDT0000 complete [ 5668.324474] LustreError: 138881:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5668.356213] LustreError: 138881:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5672.695215] LustreError: 138861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785939373 with bad export cookie 14202760842780895705 [ 5672.710763] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5672.739423] LustreError: 138861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5673.614564] Lustre: server umount lustre-MDT0001 complete [ 5710.930508] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [ 5715.214405] Key type lgssc unregistered [ 5715.723325] LNet: 145329:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5715.742116] LNetError: 145329:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5716.772810] LNet: Removed LNI 192.168.203.120@tcp [ 5718.358890] Key type .llcrypt unregistered [ 5718.362846] Key type ._llcrypt unregistered [ 5771.645977] Key type ._llcrypt registered [ 5771.653415] Key type .llcrypt registered [ 5771.791206] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [ 5794.057741] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 5795.816396] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5795.863640] alg: No test for adler32 (adler32-zlib) [ 5797.137014] Lustre: Lustre: Build Version: 2.17.56_50_g82e980f [ 5797.604066] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [ 5799.320515] Key type lgssc registered [ 5800.857534] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5852.826883] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5852.885037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5854.597379] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5854.748177] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5855.002921] Lustre: lustre-MDT0000: new disk, initializing [ 5855.185381] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5855.223117] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5862.081827] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 5876.685406] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5876.816917] Lustre: 149488:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5876.890788] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5876.899585] Lustre: Skipped 1 previous similar message [ 5877.110967] Lustre: lustre-MDT0001: new disk, initializing [ 5877.368633] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5877.435583] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5877.454203] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5884.484817] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 5899.835967] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5909.374626] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5920.771136] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5921.193305] Lustre: lustre-OST0000: new disk, initializing [ 5921.201072] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5921.213454] Lustre: 151462:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5921.379445] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5930.586097] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5930.607679] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5930.784060] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 5930.823033] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5941.745454] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5947.738942] Lustre: Failing over lustre-OST0000 [ 5948.148212] Lustre: server umount lustre-OST0000 complete [ 5965.286318] Lustre: 147712:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785939649/real 1785939649] req@00000000afb56d51 x1872693295093248/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1785939665 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5965.313726] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5967.848441] Lustre: 147709:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785939652/real 1785939652] req@00000000f9b443bb x1872693295094144/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1785939668 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5967.884856] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5970.103764] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5970.574045] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5970.613130] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 5972.448494] Lustre: 147709:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785939657/real 1785939657] req@00000000fec4f63e x1872693295095552/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1785939673 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5972.494561] Lustre: 147709:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 5974.647620] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 5977.699745] Lustre: 147710:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785939662/real 1785939662] req@00000000348739f2 x1872693295096704/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1785939678 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5977.741166] Lustre: 147710:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 5977.813321] Lustre: lustre-OST0000: Recovery over after 0:03, of 3 clients 3 recovered and 0 were evicted. [ 5977.816195] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5980.260973] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 5985.131340] Lustre: Failing over lustre-MDT0000 [ 5985.174281] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.20@tcp (stopping) [ 5985.261282] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5985.280695] Lustre: Skipped 1 previous similar message [ 5987.584748] Lustre: server umount lustre-MDT0000 complete [ 5995.385673] LustreError: 150623:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5995.427363] LustreError: 150623:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6000.513246] LustreError: 149496:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6000.553879] LustreError: 149496:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6005.632036] LustreError: 151802:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6005.667378] LustreError: 151802:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6005.733210] Lustre: 147711:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785939690/real 1785939690] req@000000008db506ce x1872693295110784/t0(0) o400->MGC192.168.203.120@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1785939706 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6005.774971] Lustre: 147711:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 6005.780660] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6010.066022] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6010.768172] LustreError: 149496:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6010.800849] LustreError: 149496:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6015.879645] LustreError: 150623:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6015.907128] LustreError: 150623:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6016.472610] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6016.589294] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6020.984253] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6021.613342] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6021.619451] Lustre: Skipped 1 previous similar message [ 6021.662415] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 6021.736854] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 6023.287891] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 6034.401194] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6034.404340] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6034.442561] Lustre: Skipped 1 previous similar message [ 6034.445158] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6034.447613] Lustre: Skipped 3 previous similar messages [ 6036.979297] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6036.991939] Lustre: Skipped 1 previous similar message [ 6040.593954] Lustre: server umount lustre-OST0000 complete [ 6047.200760] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6047.210479] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6047.215601] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6047.257987] Lustre: Skipped 2 previous similar messages [ 6052.329804] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6052.341318] Lustre: Skipped 2 previous similar messages [ 6053.918963] Lustre: server umount lustre-MDT0000 complete [ 6057.449654] LustreError: 151802:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6057.487165] LustreError: 151802:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 6060.184886] LustreError: 149481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785939760 with bad export cookie 11499455127104993592 [ 6060.193848] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6060.201482] LustreError: 149481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6061.227882] Lustre: server umount lustre-MDT0001 complete [ 6082.746332] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [ 6087.337234] Key type lgssc unregistered [ 6087.807592] LNet: 155151:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6087.833200] LNetError: 155151:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6088.870861] LNet: Removed LNI 192.168.203.120@tcp [ 6090.358230] Key type .llcrypt unregistered [ 6090.359586] Key type ._llcrypt unregistered [ 6138.917932] Key type ._llcrypt registered [ 6138.919105] Key type .llcrypt registered [ 6139.063817] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 6139.945024] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6140.167128] alg: No test for adler32 (adler32-zlib) [ 6141.351960] Lustre: Lustre: Build Version: 2.17.56_50_g82e980f [ 6141.708106] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [ 6143.568495] Key type lgssc registered [ 6145.289300] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6159.571939] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6159.639885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6162.081645] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6170.684906] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 6171.680984] LustreError: 156677:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6171.712908] LustreError: 156677:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6176.736868] LustreError: 156676:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6181.859766] LustreError: 156677:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6185.157474] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6186.976720] LustreError: 156676:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6189.318544] LustreError: 156705:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 6189.327428] LustreError: 156705:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 6189.338956] LustreError: 156705:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 27, retries 0, failed: rc = -5 [ 6189.669514] Lustre: server umount lustre-MDT0000 complete [ 6209.480909] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [ 6220.745888] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 6273.578696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6273.769437] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6273.801569] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6273.869329] Lustre: lustre-MDT0000: new disk, initializing [ 6274.043898] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6274.092246] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6281.484799] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 6297.282233] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6297.396626] Lustre: 160013:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6297.440873] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6297.443492] Lustre: Skipped 1 previous similar message [ 6297.498869] Lustre: lustre-MDT0001: new disk, initializing [ 6297.573635] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6297.609921] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6297.613463] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6303.436590] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 6317.163696] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6325.364577] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6334.511638] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6335.236385] Lustre: lustre-OST0000: new disk, initializing [ 6335.244592] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6335.265223] Lustre: 161981:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6335.555911] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6344.850982] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 6345.349877] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6345.365639] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6345.606515] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6360.762314] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6367.005530] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 6369.164771] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 10:27:47 (1785940067) [ 6712.290873] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6712.306672] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6712.345809] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6714.848784] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6714.849718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6714.872770] Lustre: Skipped 1 previous similar message [ 6714.908925] Lustre: Skipped 2 previous similar messages [ 6717.281147] Lustre: server umount lustre-MDT0000 complete [ 6719.970038] LustreError: 162172:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6720.000799] LustreError: 162172:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6723.892246] LustreError: 160004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785940424 with bad export cookie 8302823010923918899 [ 6723.893832] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6723.924550] LustreError: 160004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6725.088265] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6725.101427] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6728.096363] LustreError: 160020:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6728.107041] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6728.135622] LustreError: 160020:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6733.281479] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6738.399155] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6739.092868] Lustre: server umount lustre-MDT0001 complete [ 6759.391932] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6759.848846] Lustre: server umount lustre-OST0000 complete [ 6773.741366] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [ 6788.068274] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 6837.889052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6838.628991] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6838.722505] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6838.952586] Lustre: lustre-MDT0000: new disk, initializing [ 6839.173351] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6839.232235] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6845.652760] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 6858.258904] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6858.427840] Lustre: 166675:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6858.552253] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6858.561549] Lustre: Skipped 1 previous similar message [ 6858.683230] Lustre: lustre-MDT0001: new disk, initializing [ 6858.781783] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6858.837544] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6858.854906] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6865.669996] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 6879.256831] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6888.277150] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6898.629401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6899.252682] Lustre: lustre-OST0000: new disk, initializing [ 6899.264115] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6899.292940] Lustre: 168647:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6899.468311] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6907.636972] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 6909.557223] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6909.575628] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6909.707293] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6920.330627] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6926.321433] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6926.708881] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6931.511934] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6931.743248] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6935.521942] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6935.522592] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6935.544432] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6935.567671] LustreError: Skipped 1 previous similar message [ 6935.577036] Lustre: Skipped 1 previous similar message [ 6940.649337] LustreError: 169077:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6940.672339] LustreError: 169077:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6940.861920] Lustre: server umount lustre-OST0000 complete [ 6945.760589] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6945.771159] Lustre: Skipped 1 previous similar message [ 6945.777299] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6951.959687] Lustre: server umount lustre-MDT0000 complete [ 6957.421396] LustreError: 166668:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785940658 with bad export cookie 8302823010924077736 [ 6957.423288] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6957.447117] LustreError: 166668:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6958.169049] Lustre: server umount lustre-MDT0001 complete [ 6970.381445] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 10:37:48 (1785940668) [ 6994.006448] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [ 7011.163761] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 7060.375292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7060.616519] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7060.639208] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7060.726770] Lustre: lustre-MDT0000: new disk, initializing [ 7060.811696] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7060.830084] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7068.439274] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 7081.449550] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7081.658837] Lustre: 173430:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7082.147555] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7088.141708] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 7099.607906] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7106.929340] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7115.764830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7116.330445] Lustre: lustre-OST0000: new disk, initializing [ 7116.342049] Lustre: Skipped 1 previous similar message [ 7116.358448] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7116.370513] Lustre: Skipped 2 previous similar messages [ 7116.377428] Lustre: 175408:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7121.510938] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7121.531989] Lustre: Skipped 1 previous similar message [ 7121.562519] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7121.839045] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7125.192149] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 7136.722913] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7152.608852] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7152.613100] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7152.631357] Lustre: Skipped 1 previous similar message [ 7152.642568] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7152.674193] Lustre: Skipped 6 previous similar messages [ 7157.130534] Lustre: server umount lustre-OST0000 complete [ 7171.350458] Lustre: server umount lustre-MDT0000 complete [ 7173.088252] LustreError: 173438:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7173.118369] LustreError: 173438:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 7177.182911] LustreError: 173421:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785940877 with bad export cookie 8302823010924078618 [ 7177.196542] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7177.199043] LustreError: 173421:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7178.426735] Lustre: server umount lustre-MDT0001 complete [ 7190.596626] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 10:41:28 (1785940888) [ 7212.764291] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [ 7227.362358] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 7273.454903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7273.864717] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7273.923252] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7274.092604] Lustre: lustre-MDT0000: new disk, initializing [ 7274.259929] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7274.270503] Lustre: Skipped 2 previous similar messages [ 7274.323090] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7279.722168] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 7293.801108] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7294.071626] Lustre: 179738:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7294.551434] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7301.981136] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 7314.028691] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7322.098344] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7331.962626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7332.482229] Lustre: 181716:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7342.488913] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 7342.760284] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7342.958044] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7356.103926] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7368.683468] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7368.684021] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7368.686563] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7368.686569] Lustre: Skipped 2 previous similar messages [ 7368.691429] Lustre: Skipped 4 previous similar messages [ 7368.718228] LustreError: Skipped 1 previous similar message [ 7371.877578] Lustre: server umount lustre-OST0000 complete [ 7383.996787] Lustre: server umount lustre-MDT0000 complete [ 7384.040410] LustreError: 179749:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7384.068695] LustreError: 179749:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7389.757448] LustreError: 179728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785941090 with bad export cookie 8302823010924079703 [ 7389.761866] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7389.770564] LustreError: 179728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7390.814649] Lustre: server umount lustre-MDT0001 complete [ 7409.706297] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [ 7414.212402] Key type lgssc unregistered [ 7414.654518] LNet: 183601:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7414.667258] LNetError: 183601:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7414.702714] LNet: Removed LNI 192.168.203.120@tcp [ 7416.099726] Key type .llcrypt unregistered [ 7416.103600] Key type ._llcrypt unregistered [ 7454.147107] Key type ._llcrypt registered [ 7454.149226] Key type .llcrypt registered [ 7454.254439] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 7455.434088] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7455.456969] alg: No test for adler32 (adler32-zlib) [ 7456.811432] Lustre: Lustre: Build Version: 2.17.56_50_g82e980f [ 7457.259929] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [ 7459.127144] Key type lgssc registered [ 7461.516388] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7473.888508] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7473.934255] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7475.602639] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7480.801070] LustreError: 185018:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7480.829284] LustreError: 185018:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7484.231177] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 7485.930274] LustreError: 185019:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7491.041373] LustreError: 185018:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7496.163771] LustreError: 185019:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7497.809489] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7498.547200] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7506.625978] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 7516.681966] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7523.280441] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7531.956296] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7532.721664] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7543.705436] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 7546.082805] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 7554.515389] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7579.115291] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7579.124507] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7579.143906] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7582.185997] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7582.186921] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7582.226188] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7583.480935] Lustre: server umount lustre-OST0000 complete [ 7590.370599] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7590.386982] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7590.405843] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7590.411177] Lustre: Skipped 1 previous similar message [ 7592.416534] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7592.420831] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7594.586641] Lustre: server umount lustre-MDT0000 complete [ 7597.536244] LustreError: 185015:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7597.557062] LustreError: 185015:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7599.908668] LustreError: 185001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785941300 with bad export cookie 18040450784862546574 [ 7599.920832] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7599.927838] LustreError: 185001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7600.609870] Lustre: server umount lustre-MDT0001 complete [ 7615.855704] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [ 7620.069806] Key type lgssc unregistered [ 7620.488220] LNet: 188580:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7620.510646] LNetError: 188580:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7620.538624] LNet: Removed LNI 192.168.203.120@tcp [ 7621.707350] Key type .llcrypt unregistered [ 7621.710242] Key type ._llcrypt unregistered [ 7653.737537] Key type ._llcrypt registered [ 7653.742579] Key type .llcrypt registered [ 7653.956141] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 7655.438618] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7655.551596] alg: No test for adler32 (adler32-zlib) [ 7656.612478] Lustre: Lustre: Build Version: 2.17.56_50_g82e980f [ 7656.780931] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [ 7658.431190] Key type lgssc registered [ 7660.037923] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7670.400811] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7670.438100] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7672.022406] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7677.036650] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 7681.506582] LustreError: 189999:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7681.524582] LustreError: 189999:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7686.641130] LustreError: 189998:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7688.277983] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7689.001598] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7694.778180] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 7703.768988] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7711.351842] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7719.785242] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7720.034438] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7728.303547] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 7740.223336] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7741.580319] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 7748.498912] Lustre: lustre-OST0000: Not available for connect from 192.168.203.20@tcp (stopping) [ 7751.141384] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7751.157592] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7751.185353] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7752.160185] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7752.163209] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7752.196697] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7752.205949] Lustre: Skipped 1 previous similar message [ 7754.645875] Lustre: server umount lustre-OST0000 complete [ 7765.320462] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8a02c9aef000.ost_server_uuid 50 [ 7767.292361] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 7786.488663] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7786.502952] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7786.527978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7786.537662] Lustre: Skipped 1 previous similar message [ 7790.468868] Lustre: server umount lustre-MDT0000 complete [ 7792.110864] LustreError: 189993:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7792.139117] LustreError: 189993:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7796.462684] LustreError: 189979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785941497 with bad export cookie 13714677567750080826 [ 7796.465737] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7796.490107] LustreError: 189979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7796.882348] Lustre: server umount lustre-MDT0001 complete [ 7807.044169] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 10:51:46 (1785941506) [ 7814.725467] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7815.361528] LustreError: 193921:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7815.458039] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7820.768125] LustreError: 193922:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7822.472739] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 7829.984441] LustreError: 193921:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7829.997235] LustreError: 193921:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7832.285334] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7832.850648] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7840.346666] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 7850.431055] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7859.928723] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7873.821833] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7874.186095] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7883.837961] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 7885.438573] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7896.348931] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7906.238870] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7906.611888] Lustre: lustre-OST0001: new disk, initializing [ 7906.618169] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7906.631787] Lustre: 196766:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7906.765237] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7915.666917] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7915.682758] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7915.890477] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7916.728712] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 7927.196875] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7931.362266] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7931.376877] Lustre: Skipped 2 previous similar messages [ 7931.380017] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7931.394969] Lustre: Skipped 2 previous similar messages [ 7937.494854] Lustre: server umount lustre-OST0000 complete [ 7938.436062] LustreError: 197072:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7945.750912] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7945.975967] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7952.617051] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7952.925490] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7972.320836] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7972.328829] LustreError: 195747:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7972.337466] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7972.338752] LustreError: Skipped 1 previous similar message [ 7972.340422] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7972.340427] Lustre: Skipped 3 previous similar messages [ 7972.432726] LustreError: 195747:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 18 previous similar messages [ 7973.881486] Lustre: server umount lustre-OST0001 complete [ 7981.538034] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7981.550385] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7981.567170] Lustre: Skipped 1 previous similar message [ 7985.124067] Lustre: server umount lustre-MDT0000 complete [ 7989.854240] LustreError: 193901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785941690 with bad export cookie 13714677567750081897 [ 7989.869869] LustreError: 189311:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000f769498e x1872695244468608/t0(0) o41->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 224/224 e 0 to 0 dl 1785941698 ref 1 fl Rpc:RESQU/200/0 rc -5/-107 job:'' uid:0 gid:0 projid:4294967295 [ 7989.875720] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7990.558159] Lustre: server umount lustre-MDT0001 complete [ 8020.151334] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 10:55:18 (1785941718) [ 8030.594175] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8031.252456] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8031.318404] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8031.909470] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8042.686694] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 8056.377889] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8056.513079] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8056.559453] Lustre: 200211:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8063.604902] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 8072.496646] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8079.201844] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8087.188704] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8087.498675] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8095.420691] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 8097.009218] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 8106.323319] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8117.200271] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8117.342694] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 8117.774479] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8117.780363] Lustre: Skipped 2 previous similar messages [ 8124.229079] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 8127.456745] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 8137.082145] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8143.281960] Lustre: lustre-OST0000: Not available for connect from 192.168.203.20@tcp (stopping) [ 8143.296672] Lustre: Skipped 4 previous similar messages [ 8143.328325] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8143.328816] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8143.335566] LustreError: Skipped 1 previous similar message [ 8143.364881] Lustre: Skipped 2 previous similar messages [ 8146.561628] Lustre: server umount lustre-OST0000 complete [ 8148.343664] LustreError: 202064:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8148.362407] LustreError: 202064:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8165.348979] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8165.362432] LustreError: Skipped 1 previous similar message [ 8166.090522] Lustre: server umount lustre-OST0001 complete [ 8175.587076] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8175.587944] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8175.590514] Lustre: Skipped 7 previous similar messages [ 8175.620029] Lustre: Skipped 3 previous similar messages [ 8177.558858] Lustre: server umount lustre-MDT0000 complete [ 8182.750180] LustreError: 200202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785941883 with bad export cookie 13714677567750083304 [ 8182.750663] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8182.768929] LustreError: 200202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 8183.635032] Lustre: server umount lustre-MDT0001 complete [ 8214.538789] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 10:58:33 (1785941913) [ 8236.154258] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [ 8248.965959] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 8299.220585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8299.610231] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8299.679230] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8299.755692] Lustre: lustre-MDT0000: new disk, initializing [ 8299.887599] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8299.917529] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8305.506491] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 8318.821750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8318.956140] Lustre: 208317:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8319.056547] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8319.070931] Lustre: Skipped 1 previous similar message [ 8319.251354] Lustre: lustre-MDT0001: new disk, initializing [ 8319.455972] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8319.488698] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8325.876870] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 8337.802108] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8345.210331] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8355.492646] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8355.965357] Lustre: lustre-OST0000: new disk, initializing [ 8355.970923] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8355.981027] Lustre: 210286:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8362.091848] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8362.111460] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8362.278335] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8364.261278] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 8374.985087] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8381.417680] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8381.748839] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8387.631661] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8388.053494] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8393.185830] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8393.188615] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8393.190115] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8393.190119] Lustre: Skipped 1 previous similar message [ 8393.209039] LustreError: Skipped 2 previous similar messages [ 8393.222713] Lustre: Skipped 1 previous similar message [ 8397.120872] Lustre: server umount lustre-OST0000 complete [ 8416.223574] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8416.766982] Lustre: server umount lustre-MDT0000 complete [ 8417.760704] LustreError: 208327:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8417.786024] LustreError: 208327:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [ 8421.427779] LustreError: 209111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785942122 with bad export cookie 13714677567750085089 [ 8421.441407] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8421.441599] LustreError: 209111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8421.993607] Lustre: server umount lustre-MDT0001 complete [ 8433.643141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8439.159580] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 8450.765522] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8457.967433] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 8468.251217] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8475.923353] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8484.264200] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8492.232692] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 8502.355283] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8508.014586] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8508.300751] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8513.293870] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8513.508637] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8516.576359] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8524.773414] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8524.780892] Lustre: Skipped 10 previous similar messages [ 8529.888932] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8530.204476] Lustre: server umount lustre-OST0000 complete [ 8538.204458] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8543.348601] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8574.753872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8575.027954] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8575.034336] Lustre: Skipped 5 previous similar messages [ 8582.396324] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 8585.260869] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8585.306433] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8585.313930] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8585.359823] Lustre: Skipped 1 previous similar message [ 8593.818461] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8596.221264] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8605.666197] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8605.681103] Lustre: Skipped 5 previous similar messages [ 8608.863796] Lustre: server umount lustre-OST0000 complete [ 8620.809676] Lustre: server umount lustre-MDT0000 complete [ 8626.955346] LustreError: 216631:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785942327 with bad export cookie 13714677567750085971 [ 8626.971723] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8626.987825] LustreError: 216631:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8628.003932] Lustre: server umount lustre-MDT0001 complete [ 8640.660473] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 11:05:39 (1785942339) [ 8648.405312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8655.728523] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 8666.544237] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8675.113593] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 8689.175686] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8700.542601] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8712.001240] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8712.246123] LustreError: 220317:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8712.317082] LustreError: 220317:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [ 8722.566301] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 8731.908329] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8738.863439] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8739.239516] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8746.587356] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8746.892579] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8754.996465] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8755.378906] Lustre: lustre-OST0001: new disk, initializing [ 8755.390086] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8755.403922] Lustre: 221868:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8765.215981] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8765.242960] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8765.546209] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8766.892955] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 8784.123622] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8790.721855] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 8791.074855] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8796.788654] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 8797.023909] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8801.776908] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8801.788071] Lustre: Skipped 6 previous similar messages [ 8801.789583] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8801.828478] LustreError: Skipped 2 previous similar messages [ 8805.566560] Lustre: server umount lustre-OST0001 complete [ 8814.163922] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 8814.482934] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 8820.400925] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 8820.778518] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 8851.587091] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8857.591502] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 8857.608571] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 8857.624292] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8860.109428] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 8869.754937] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8871.713523] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8878.063083] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8878.079971] Lustre: Skipped 5 previous similar messages [ 8880.686299] Lustre: server umount lustre-OST0001 complete [ 8906.789351] Lustre: server umount lustre-OST0000 complete [ 8918.126742] Lustre: server umount lustre-MDT0000 complete [ 8923.446370] LustreError: 218472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785942624 with bad export cookie 13714677567750086937 [ 8923.461118] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8930.175156] Lustre: server umount lustre-MDT0001 complete [ 8961.549730] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 11:11:00 (1785942660) [ 8970.319635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8970.791608] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8970.836601] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8977.326873] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 8988.882402] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8989.146637] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8989.221836] Lustre: 227117:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8996.983114] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 9011.841531] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9019.948504] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9029.486071] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9029.769788] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9037.629112] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 9048.158663] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9059.497152] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9059.599344] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9071.221281] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 9084.312278] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9087.479598] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [ 9096.004565] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 9096.441441] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9102.399288] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 9102.566254] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9110.924547] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8a02cb1d5800.ost_server_uuid 50 [ 9114.132945] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8a02cb1d5800.ost_server_uuid in IDLE state after 0 sec [ 9115.671704] Lustre: Permanently deactivating lustre-OST0001 [ 9115.715474] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 9121.438262] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 9123.010621] Lustre: Permanently reactivating lustre-OST0001 [ 9123.031643] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 9123.035714] Lustre: Skipped 2 previous similar messages [ 9128.173631] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 9128.190573] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 9128.216184] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9128.218590] Lustre: Skipped 1 previous similar message [ 9128.285657] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 9134.949396] Lustre: server umount lustre-OST0001 complete [ 9146.742204] Lustre: server umount lustre-OST0000 complete [ 9166.304592] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9166.758646] Lustre: server umount lustre-MDT0000 complete [ 9172.158901] LustreError: 227109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785942872 with bad export cookie 13714677567750088225 [ 9172.179240] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9172.180623] LustreError: 227109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9173.451033] Lustre: server umount lustre-MDT0001 complete [ 9207.785486] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 11:15:06 (1785942906) [ 9217.301780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9217.738133] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9217.839569] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9217.848459] Lustre: Skipped 2 previous similar messages [ 9218.156457] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9218.165189] Lustre: Skipped 9 previous similar messages [ 9225.414037] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 9238.645373] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9238.833401] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9238.887860] Lustre: 233831:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9247.092160] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 9262.165823] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9270.446599] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9279.317932] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9279.671586] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9279.735073] Lustre: Permanently deactivating lustre-OST0000 [ 9279.753760] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 9287.950691] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 9289.275222] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 9289.289464] Lustre: Skipped 1 previous similar message [ 9295.973543] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9306.415240] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9306.457188] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9314.968649] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 9315.010814] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 9325.695644] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9328.076265] Lustre: Permanently reactivating lustre-OST0000 [ 9328.110202] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 9328.122935] Lustre: Skipped 2 previous similar messages [ 9336.437859] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 9336.453288] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 48 seconds. I think it's dead, and I am evicting it. exp 00000000d7b858c9, cur 1785943037 deadline 1785943019 last 1785942989 [ 9336.505471] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 9346.029595] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9346.032508] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9346.042715] LustreError: Skipped 5 previous similar messages [ 9346.048799] Lustre: Skipped 19 previous similar messages [ 9348.686897] Lustre: server umount lustre-OST0001 complete [ 9351.142896] LustreError: 235685:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9351.153790] LustreError: 235685:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 45 previous similar messages [ 9359.618541] Lustre: server umount lustre-OST0000 complete [ 9370.680293] Lustre: server umount lustre-MDT0000 complete [ 9375.907420] LustreError: 233822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785943076 with bad export cookie 13714677567750090024 [ 9375.909372] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9375.919377] LustreError: 233822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9376.865976] Lustre: server umount lustre-MDT0001 complete [ 9391.272846] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [ 9394.917615] Key type lgssc unregistered [ 9395.318740] LNet: 238719:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9395.330642] LNetError: 238719:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9395.352201] LNet: Removed LNI 192.168.203.120@tcp [ 9396.571797] Key type .llcrypt unregistered [ 9396.577548] Key type ._llcrypt unregistered [ 9424.599656] Key type ._llcrypt registered [ 9424.601767] Key type .llcrypt registered [ 9424.706650] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 9425.728645] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9425.919727] alg: No test for adler32 (adler32-zlib) [ 9427.040678] Lustre: Lustre: Build Version: 2.17.56_50_g82e980f [ 9427.318556] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [ 9428.951991] Key type lgssc registered [ 9430.583632] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9442.517422] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9442.535522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9444.273635] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9450.530440] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 9453.549825] LustreError: 240189:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9453.573905] LustreError: 240189:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9458.661183] LustreError: 240188:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9462.266858] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9462.447879] Lustre: Found index 1 for lustre-MDT0001, updating log [ 9462.490439] Lustre: 240178:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9462.512916] Lustre: Permanently deactivating lustre-MDT0001 [ 9462.540935] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 9462.972657] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9468.864618] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 9469.485559] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 9478.915328] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9487.120077] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9497.028380] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9497.634283] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9505.400885] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 9507.882799] LustreError: 242031:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9512.936931] LustreError: 242031:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9515.011737] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 9515.197709] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9520.103170] LustreError: 242029:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9520.128728] LustreError: 242029:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9523.957667] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9524.210051] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9529.328536] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 9529.341163] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 9531.262636] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 9539.869709] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9544.084311] Lustre: Permanently reactivating lustre-MDT0001 [ 9544.120491] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 9544.127254] Lustre: Skipped 2 previous similar messages [ 9548.139945] Lustre: lustre-MDT0001: Client fe849bcc-19e9-4cf5-a6bf-1de7a2fecad7 (at 192.168.203.20@tcp) reconnecting [ 9551.020533] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9551.042797] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 9551.054651] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 9551.071393] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 9555.562750] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9556.974068] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9563.514930] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 9563.681362] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 9569.820583] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 9570.016296] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 9573.822136] Lustre: Permanently deactivating lustre-MDT0001 [ 9573.848037] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 9573.851479] Lustre: Skipped 2 previous similar messages [ 9581.423339] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 9595.896031] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9595.899517] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9595.915245] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9595.938304] Lustre: Skipped 2 previous similar messages [ 9600.998886] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9601.024920] Lustre: Skipped 6 previous similar messages [ 9603.090500] Lustre: server umount lustre-MDT0000 complete [ 9606.114318] LustreError: 240188:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9606.160153] LustreError: 240188:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9609.471137] LustreError: 240169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785943310 with bad export cookie 13080726821809909946 [ 9609.492313] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9610.401517] Lustre: server umount lustre-MDT0001 complete [ 9616.658143] Lustre: server umount lustre-OST0000 complete [ 9633.232938] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 11:22:12 (1785943332) [ 9645.900048] Lustre: server umount lustre-OST0001 complete [ 9656.235099] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [ 9677.245167] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 9725.148773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9725.490787] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9725.496961] Lustre: Skipped 2 previous similar messages [ 9725.512332] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9725.563166] Lustre: lustre-MDT0000: new disk, initializing [ 9725.614334] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9725.651782] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9732.631606] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 9745.318290] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9745.491766] Lustre: 248527:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9745.602291] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9745.610178] Lustre: Skipped 1 previous similar message [ 9745.773234] Lustre: lustre-MDT0001: new disk, initializing [ 9745.971519] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9746.035199] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9746.054448] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9753.058293] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 9767.451853] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9776.907017] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9786.548412] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9786.971436] Lustre: lustre-OST0000: new disk, initializing [ 9786.975890] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9786.989383] Lustre: 250499:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9787.109043] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9793.630982] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9793.651537] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9793.803405] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9794.531382] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 9808.333281] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9811.060641] LustreError: 248535:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 9821.107172] LustreError: 248535:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 9821.142052] LustreError: 250983:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 9821.897841] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9822.148982] Lustre: lustre-OST0001: new disk, initializing [ 9822.156583] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9822.186837] Lustre: 251615:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9822.348841] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9829.106042] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9829.138226] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9829.317267] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9831.226322] LustreError: 250983:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 9831.992593] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 9845.879838] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9848.706294] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 9854.970615] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9854.974037] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9854.988198] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9854.998671] Lustre: Skipped 2 previous similar messages [ 9859.365871] Lustre: server umount lustre-OST0001 complete [ 9860.135366] LustreError: 250481:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9860.157184] LustreError: 250481:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9881.056213] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9881.058566] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9881.072273] LustreError: Skipped 1 previous similar message [ 9881.075836] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9881.075843] Lustre: Skipped 1 previous similar message [ 9881.141276] Lustre: Skipped 1 previous similar message [ 9885.670193] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9885.677117] Lustre: Skipped 2 previous similar messages [ 9895.391295] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9896.058503] Lustre: server umount lustre-OST0000 complete [ 9904.608854] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9904.624751] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9904.665836] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9904.674759] Lustre: Skipped 2 previous similar messages [ 9909.784184] Lustre: server umount lustre-MDT0000 complete [ 9911.271000] LustreError: 248536:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9911.306611] LustreError: 248536:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [ 9915.493735] LustreError: 248521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785943616 with bad export cookie 13080726821809912543 [ 9915.496113] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9915.506804] LustreError: 248521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9922.105162] Lustre: server umount lustre-MDT0001 complete [ 9937.472081] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [ 9941.778462] Key type lgssc unregistered [ 9942.179701] LNet: 253657:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9942.199179] LNetError: 253657:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9943.271895] LNet: Removed LNI 192.168.203.120@tcp [ 9944.755049] Key type .llcrypt unregistered [ 9944.759439] Key type ._llcrypt unregistered [10008.486454] Key type ._llcrypt registered [10008.489636] Key type .llcrypt registered [10008.643082] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [10010.541182] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10010.672449] alg: No test for adler32 (adler32-zlib) [10011.935986] Lustre: Lustre: Build Version: 2.17.56_50_g82e980f [10012.223180] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [10013.976069] Key type lgssc registered [10015.760739] Lustre: Echo OBD driver; http://www.lustre.org/ [10026.460958] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10026.490694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10027.926538] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10027.989800] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10028.240168] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10034.164815] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [10045.362438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10045.545939] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10045.608731] Lustre: 255871:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10046.084911] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10053.540588] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [10065.025185] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10076.240149] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10093.859229] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10094.409089] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10095.091025] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10102.221796] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [10107.196327] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [10115.171644] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10176.816815] Lustre: server umount lustre-OST0000 complete [10182.762109] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [10196.447081] Lustre: 255192:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785943879/real 1785943879] req@00000000be310319 x1872697713806720/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1785943895 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10196.478095] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10199.456686] Lustre: 255194:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785943884/real 1785943884] req@00000000992ded54 x1872697713808384/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1785943900 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10199.489483] Lustre: 255194:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [10199.991776] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10200.503092] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10204.575431] Lustre: 255194:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785943889/real 1785943889] req@0000000024180f61 x1872697713809408/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1785943905 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10204.618886] Lustre: 255194:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [10206.645162] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [10206.660801] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [10206.672772] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [10208.545527] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [10220.941898] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10268.138312] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10268.142323] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10268.144335] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10268.153345] Lustre: Skipped 2 previous similar messages [10271.193096] Lustre: server umount lustre-OST0000 complete [10276.320996] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10276.325345] LustreError: Skipped 1 previous similar message [10276.330373] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10276.339987] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10276.342229] Lustre: Skipped 1 previous similar message [10278.375700] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10278.376041] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10278.409608] Lustre: Skipped 1 previous similar message [10283.488186] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10287.588834] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10287.595790] Lustre: Skipped 2 previous similar messages [10290.143148] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10290.679360] Lustre: server umount lustre-MDT0000 complete [10295.411972] LustreError: 255864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785943996 with bad export cookie 8567698993399897281 [10295.421816] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10295.425484] LustreError: 255864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10296.215846] Lustre: server umount lustre-MDT0001 complete [10311.207580] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [10315.901338] Key type lgssc unregistered [10316.491531] LNet: 261308:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10316.502801] LNetError: 261308:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10316.535375] LNet: Removed LNI 192.168.203.120@tcp [10317.916484] Key type .llcrypt unregistered [10317.924733] Key type ._llcrypt unregistered [10350.066712] Key type ._llcrypt registered [10350.086585] Key type .llcrypt registered [10350.219958] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [10351.624990] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10351.654716] alg: No test for adler32 (adler32-zlib) [10352.960559] Lustre: Lustre: Build Version: 2.17.56_50_g82e980f [10353.252092] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [10354.943160] Key type lgssc registered [10356.629887] Lustre: Echo OBD driver; http://www.lustre.org/ [10367.545216] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10367.623370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10369.592705] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10378.419253] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [10379.746544] LustreError: 262727:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10379.806596] LustreError: 262727:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10380.770339] LustreError: 262728:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10385.907954] LustreError: 262727:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10391.016753] LustreError: 262728:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10393.446834] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10394.280608] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10403.917804] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [10416.405229] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10425.224636] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10435.833574] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10436.517199] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10440.637087] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [10447.594310] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [10458.551792] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10489.825523] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10489.835695] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10489.865656] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10491.872986] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10491.878483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10491.893939] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10491.942886] Lustre: Skipped 1 previous similar message [10495.991950] Lustre: server umount lustre-OST0000 complete [10501.602986] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10501.617937] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10501.646238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10506.801302] Lustre: server umount lustre-MDT0000 complete [10507.232526] LustreError: 263456:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10507.245325] LustreError: 263456:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10512.147503] LustreError: 265998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785944212 with bad export cookie 1599726066216719502 [10512.169454] LustreError: 265998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10512.172032] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10513.298059] Lustre: server umount lustre-MDT0001 complete [10529.935367] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [10534.064542] Key type lgssc unregistered [10534.542640] LNet: 267045:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10534.556970] LNetError: 267045:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10535.589675] LNet: Removed LNI 192.168.203.120@tcp [10536.889815] Key type .llcrypt unregistered [10536.891362] Key type ._llcrypt unregistered [10568.018485] Key type ._llcrypt registered [10568.032773] Key type .llcrypt registered [10568.416763] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [10572.467162] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10572.609412] alg: No test for adler32 (adler32-zlib) [10574.126243] Lustre: Lustre: Build Version: 2.17.56_50_g82e980f [10574.446931] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [10576.183154] Key type lgssc registered [10578.688423] Lustre: Echo OBD driver; http://www.lustre.org/ [10592.955906] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10592.997541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10594.896255] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10602.167189] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [10605.028993] LustreError: 268289:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10605.060416] LustreError: 268289:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10610.146338] LustreError: 268288:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10613.367927] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10614.031591] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10619.610083] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [10628.520160] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10640.731301] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10654.927906] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10655.606240] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10663.992783] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [10664.816375] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10676.562329] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10699.329862] Lustre: server umount lustre-OST0000 complete [10711.027046] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10711.042777] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10711.043407] Lustre: Skipped 1 previous similar message [10711.103898] Lustre: Skipped 1 previous similar message [10716.129767] Lustre: 267596:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785944400/real 1785944400] req@00000000113d5252 x1872698304140032/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1785944416 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10716.130730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10716.167608] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10716.206426] Lustre: Skipped 1 previous similar message [10717.151306] Lustre: 267598:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785944401/real 1785944401] req@00000000ffee0284 x1872698304140800/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1785944417 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10718.394607] Lustre: server umount lustre-MDT0000 complete [10721.248624] LustreError: 268288:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10721.265209] LustreError: 268288:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10722.336878] Lustre: 267596:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785944406/real 1785944406] req@000000001735fcda x1872698304142080/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1785944422 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10723.414357] LustreError: 268269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785944424 with bad export cookie 10934436168774083176 [10723.447471] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10724.573321] Lustre: server umount lustre-MDT0001 complete [10743.917256] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [10748.670410] Key type lgssc unregistered [10749.194537] LNet: 271998:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10749.197992] LNetError: 271998:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10749.239844] LNet: Removed LNI 192.168.203.120@tcp [10751.432064] Key type .llcrypt unregistered [10751.436822] Key type ._llcrypt unregistered [10784.837107] Key type ._llcrypt registered [10784.845436] Key type .llcrypt registered [10784.964387] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [10786.548373] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10786.571062] alg: No test for adler32 (adler32-zlib) [10787.748450] Lustre: Lustre: Build Version: 2.17.56_50_g82e980f [10788.095956] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [10789.751172] Key type lgssc registered [10791.160363] Lustre: Echo OBD driver; http://www.lustre.org/ [10801.682534] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10801.714268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10803.442250] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10809.496458] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [10812.907758] LustreError: 273416:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10812.928531] LustreError: 273416:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10818.038765] LustreError: 273416:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10819.803655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10820.284401] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10826.969972] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [10837.538435] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10845.023110] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10853.529874] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10854.188215] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10862.570282] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [10872.693125] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [10874.404425] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10910.687930] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10910.694266] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10910.709041] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10913.761296] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10913.764418] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10913.785780] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10913.850283] Lustre: Skipped 1 previous similar message [10917.432358] Lustre: server umount lustre-OST0000 complete [10927.584611] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10927.590306] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10927.631553] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10931.523328] Lustre: server umount lustre-MDT0000 complete [10934.265634] LustreError: 273411:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10934.314665] LustreError: 273411:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10938.413851] LustreError: 273416:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10938.463547] LustreError: 273416:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10939.612468] LustreError: 273396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785944640 with bad export cookie 8262019481073423744 [10939.615103] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10939.629273] LustreError: 273396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10940.100621] Lustre: server umount lustre-MDT0001 complete [10962.225781] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [10967.308730] Key type lgssc unregistered [10967.809696] LNet: 277754:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10967.840308] LNetError: 277754:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10967.864594] LNet: Removed LNI 192.168.203.120@tcp [10969.296661] Key type .llcrypt unregistered [10969.298432] Key type ._llcrypt unregistered [10997.061286] Key type ._llcrypt registered [10997.066854] Key type .llcrypt registered [10997.294460] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [10998.603392] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10998.638256] alg: No test for adler32 (adler32-zlib) [11000.539199] Lustre: Lustre: Build Version: 2.17.56_50_g82e980f [11000.869507] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [11002.663245] Key type lgssc registered [11004.828519] Lustre: Echo OBD driver; http://www.lustre.org/ [11018.849000] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11018.867810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11020.730986] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11027.613373] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [11031.020050] LustreError: 278994:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11031.065249] LustreError: 278994:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11036.130681] LustreError: 278996:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11039.673047] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11040.026827] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11046.005480] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [11056.009376] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11065.577582] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11077.349634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11077.997711] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11088.088060] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [11092.388342] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [11099.254513] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11116.000460] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11116.008941] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11116.036518] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11118.052111] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11118.056722] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11118.065316] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11118.129145] Lustre: Skipped 1 previous similar message [11120.651543] Lustre: server umount lustre-OST0000 complete [11128.294977] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11128.300599] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11128.312127] Lustre: Skipped 1 previous similar message [11128.339084] Lustre: Skipped 1 previous similar message [11133.431114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11133.453117] Lustre: Skipped 1 previous similar message [11135.074989] Lustre: server umount lustre-MDT0000 complete [11138.545062] LustreError: 278997:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11138.561605] LustreError: 278997:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11140.000069] LustreError: 280865:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785944840 with bad export cookie 1848438912100912836 [11140.003776] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11140.016303] LustreError: 280865:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11140.729290] Lustre: server umount lustre-MDT0001 complete [11153.870564] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [11157.888896] Key type lgssc unregistered [11158.405589] LNet: 282703:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11158.423152] LNetError: 282703:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11158.449151] LNet: Removed LNI 192.168.203.120@tcp [11159.931921] Key type .llcrypt unregistered [11159.935441] Key type ._llcrypt unregistered [11174.477169] Key type ._llcrypt registered [11174.481537] Key type .llcrypt registered [11174.633355] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [11200.191312] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [11219.957714] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [11221.037959] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11221.174094] alg: No test for adler32 (adler32-zlib) [11222.240812] Lustre: Lustre: Build Version: 2.17.56_50_g82e980f [11222.443546] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [11224.347407] Key type lgssc registered [11226.813943] Lustre: Echo OBD driver; http://www.lustre.org/ [11277.756926] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11277.790399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11279.149965] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11279.214944] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11279.358694] Lustre: lustre-MDT0000: new disk, initializing [11279.505892] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11279.524509] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11284.609353] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [11297.990588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11298.164628] Lustre: 286269:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11298.233920] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11298.239774] Lustre: Skipped 1 previous similar message [11298.356432] Lustre: lustre-MDT0001: new disk, initializing [11298.515531] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11298.590036] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11298.604990] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11305.005543] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [11317.337279] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11326.043713] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11336.518807] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11337.294129] Lustre: lustre-OST0000: new disk, initializing [11337.312496] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11337.342336] Lustre: 288242:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11337.562095] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11345.486708] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11345.506285] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11345.661939] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11348.640301] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [11363.031665] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11370.922438] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11371.533849] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11379.230164] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11379.616343] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11386.848883] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11386.859739] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11386.895922] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11388.862174] Lustre: server umount lustre-OST0000 complete [11396.064623] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11396.071636] LustreError: Skipped 1 previous similar message [11396.080706] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11396.100337] Lustre: Skipped 1 previous similar message [11396.110101] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11396.112542] Lustre: Skipped 1 previous similar message [11397.091423] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11397.115519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11397.121715] Lustre: Skipped 1 previous similar message [11402.208683] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11402.213205] Lustre: Skipped 1 previous similar message [11407.327363] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11407.331505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11407.389879] Lustre: Skipped 1 previous similar message [11407.918491] Lustre: server umount lustre-MDT0000 complete [11413.174030] LustreError: 288241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785945113 with bad export cookie 9967209318073956488 [11413.189349] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11413.220200] LustreError: 288241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11413.620392] Lustre: server umount lustre-MDT0001 complete [11427.151461] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 11:52:04 (1785945124) [11438.626393] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11439.115360] LustreError: 290658:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11439.139880] LustreError: 290658:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11439.221017] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11444.712418] LustreError: 290659:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11445.761743] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [11448.807770] LustreError: 290658:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11453.924221] LustreError: 290659:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11459.039996] LustreError: 290658:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11459.117139] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11460.158731] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11470.694343] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [11486.203198] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11495.744206] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11505.613182] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11506.125531] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11515.031133] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [11525.521594] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11545.056126] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11545.061863] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11545.072346] LustreError: Skipped 1 previous similar message [11545.078273] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11545.098579] Lustre: Skipped 1 previous similar message [11545.130064] Lustre: Skipped 1 previous similar message [11547.312485] Lustre: server umount lustre-OST0000 complete [11552.225806] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11552.234630] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11558.257250] Lustre: server umount lustre-MDT0000 complete [11560.417568] LustreError: 290653:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11560.438514] LustreError: 290653:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11562.410325] LustreError: 290638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785945263 with bad export cookie 9967209318073957370 [11562.424692] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11562.426931] LustreError: 290638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11562.973324] Lustre: server umount lustre-MDT0001 complete [11579.334726] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [11583.179930] Key type lgssc unregistered [11583.677713] LNet: 294223:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11583.687122] LNetError: 294223:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11583.728103] LNet: Removed LNI 192.168.203.120@tcp [11585.181142] Key type .llcrypt unregistered [11585.192066] Key type ._llcrypt unregistered [11620.128302] Key type ._llcrypt registered [11620.134114] Key type .llcrypt registered [11620.324261] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [11641.760794] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [11643.922167] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11643.955539] alg: No test for adler32 (adler32-zlib) [11645.510583] Lustre: Lustre: Build Version: 2.17.56_50_g82e980f [11645.954630] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [11647.839156] Key type lgssc registered [11650.308100] Lustre: Echo OBD driver; http://www.lustre.org/ [11720.300856] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11720.346137] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11721.676468] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11721.714147] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11721.888692] Lustre: lustre-MDT0000: new disk, initializing [11721.991415] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11722.008304] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11729.586262] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [11743.402578] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11743.610148] Lustre: 298060:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11743.737075] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11743.743371] Lustre: Skipped 1 previous similar message [11743.888744] Lustre: lustre-MDT0001: new disk, initializing [11744.088613] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11744.143111] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11744.155545] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11752.083808] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [11766.963554] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11775.149630] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11786.917853] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11787.351756] Lustre: lustre-OST2710: new disk, initializing [11787.357196] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [11787.367193] Lustre: 300032:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11787.505527] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [11796.658875] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [11797.140644] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [11797.148483] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [11797.229772] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [11809.210098] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11818.405986] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11818.775884] Lustre: lustre-OST03e8: new disk, initializing [11818.792859] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [11818.809690] Lustre: 301096:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11818.990050] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [11827.845562] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [11827.863901] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [11828.026127] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [11828.641495] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [11837.657849] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11849.015710] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [11849.551181] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11860.185974] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [11860.445903] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11868.152853] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [11868.559421] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11876.044876] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [11876.528070] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11884.517386] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11884.526485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11884.564725] Lustre: Skipped 1 previous similar message [11889.649698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11889.660057] Lustre: Skipped 5 previous similar messages [11891.622946] Lustre: server umount lustre-MDT0000 complete [11896.387376] LustreError: 298050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785945596 with bad export cookie 4756929005485014158 [11896.397837] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11897.032538] Lustre: server umount lustre-MDT0001 complete [11913.986478] Lustre: server umount lustre-OST2710 complete [11916.255095] Lustre: 295853:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785945600/real 1785945600] req@00000000c90e021d x1872699427211264/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785945616 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11916.285262] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11916.307363] Lustre: Skipped 2 previous similar messages [11918.303151] Lustre: 295851:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785945602/real 1785945602] req@000000009eda09d7 x1872699427211520/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785945618 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11919.835851] Lustre: server umount lustre-OST03e8 complete [11946.214270] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [11968.073657] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [12019.319084] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12019.661436] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12019.708968] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12019.849236] Lustre: lustre-MDT0000: new disk, initializing [12019.965070] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12019.986869] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12026.732785] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [12041.669212] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12041.861274] Lustre: 306385:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12041.941229] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12041.944976] Lustre: Skipped 1 previous similar message [12042.072210] Lustre: lustre-MDT0001: new disk, initializing [12042.210422] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12042.296378] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12042.318753] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12048.481933] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [12061.393517] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12068.349992] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12077.450677] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12078.387336] Lustre: lustre-OST0000: new disk, initializing [12078.405842] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12078.426896] Lustre: 308353:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12078.684127] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12086.420440] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12086.449138] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12086.666747] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12090.818954] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [12106.397457] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12114.338379] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12114.693017] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12121.411479] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12121.661368] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12127.713096] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12127.725042] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12127.728746] LustreError: Skipped 1 previous similar message [12127.741102] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12127.773732] Lustre: Skipped 1 previous similar message [12127.827748] Lustre: Skipped 1 previous similar message [12132.850129] LustreError: 308616:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12133.003751] LustreError: 308616:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [12133.922758] Lustre: server umount lustre-OST0000 complete [12143.104485] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12143.128768] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12143.139042] Lustre: Skipped 1 previous similar message [12143.184334] Lustre: Skipped 1 previous similar message [12148.197110] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12148.212067] Lustre: Skipped 1 previous similar message [12156.384714] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12157.136468] Lustre: server umount lustre-MDT0000 complete [12157.446087] LustreError: 306392:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12157.495495] LustreError: 306392:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12162.544873] LustreError: 306397:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12162.583794] LustreError: 306397:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12163.231042] LustreError: 306379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785945863 with bad export cookie 4756929005485015852 [12163.259342] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12163.814502] Lustre: server umount lustre-MDT0001 complete [12176.255249] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 12:04:34 (1785945874) [12177.892102] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [12180.304488] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 12:04:38 (1785945878) [12187.709403] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [12235.493763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12235.998716] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12236.070877] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12236.450539] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12242.971707] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [12255.799399] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12255.982820] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12256.043167] Lustre: 312220:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12263.045897] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [12289.519798] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12289.994254] Lustre: Denying initial registration attempt from nid 192.168.203.120@tcp, specified as failover [12290.007459] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [12290.022511] LustreError: 313676:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [12290.033410] LustreError: 313676:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [12290.044901] LustreError: 313676:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [12290.199424] Lustre: server umount lustre-OST0000 complete [12290.201543] LustreError: 313676:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -99 [12301.793615] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12301.808972] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12301.815777] Lustre: Skipped 3 previous similar messages [12306.918575] LustreError: 312232:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12307.190595] Lustre: server umount lustre-MDT0000 complete [12311.761832] LustreError: 312213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785946012 with bad export cookie 4756929005485016734 [12311.768507] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12311.769886] LustreError: 312213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [12312.166410] Lustre: server umount lustre-MDT0001 complete [12328.062821] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [12332.652135] Key type lgssc unregistered [12333.532793] LNet: 314803:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12333.550543] LNetError: 314803:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12333.617480] LNet: Removed LNI 192.168.203.120@tcp [12335.753858] Key type .llcrypt unregistered [12335.760944] Key type ._llcrypt unregistered [12358.329429] Key type ._llcrypt registered [12358.334937] Key type .llcrypt registered [12358.480235] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [12360.459164] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12360.921182] alg: No test for adler32 (adler32-zlib) [12362.282898] Lustre: Lustre: Build Version: 2.17.56_50_g82e980f [12362.729720] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [12364.567150] Key type lgssc registered [12366.551707] Lustre: Echo OBD driver; http://www.lustre.org/ [12427.532226] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [12440.032178] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12440.091848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12441.369728] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12441.397385] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12441.553667] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12447.093334] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [12458.274900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12458.454941] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12458.529586] Lustre: 317588:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12459.075367] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12466.045495] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [12478.266664] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12487.968829] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12512.175715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12512.374435] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [12512.833833] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12524.813883] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [12536.956945] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12543.968021] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12543.969422] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12543.975218] LustreError: Skipped 1 previous similar message [12543.975725] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12544.028694] Lustre: Skipped 1 previous similar message [12547.050846] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12547.070500] Lustre: Skipped 2 previous similar messages [12552.187251] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12552.203107] Lustre: Skipped 1 previous similar message [12554.209796] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12554.730638] Lustre: server umount lustre-OST0000 complete [12561.376386] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12561.382552] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12561.425880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12562.400763] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12566.037855] Lustre: server umount lustre-MDT0000 complete [12571.731447] LustreError: 317580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785946272 with bad export cookie 10716274620157653240 [12571.735022] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12571.743162] LustreError: 317580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12572.602495] Lustre: server umount lustre-MDT0001 complete [12592.620863] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [12596.391139] Key type lgssc unregistered [12596.808087] LNet: 321195:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12596.819405] LNetError: 321195:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12596.848069] LNet: Removed LNI 192.168.203.120@tcp [12598.037636] Key type .llcrypt unregistered [12598.039773] Key type ._llcrypt unregistered [12638.141932] Key type ._llcrypt registered [12638.145645] Key type .llcrypt registered [12638.316261] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [12640.531514] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12640.560831] alg: No test for adler32 (adler32-zlib) [12641.939658] Lustre: Lustre: Build Version: 2.17.56_50_g82e980f [12642.419116] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [12644.287134] Key type lgssc registered [12646.260391] Lustre: Echo OBD driver; http://www.lustre.org/ [12657.744972] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12657.778557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12659.477362] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12665.922394] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [12669.922246] LustreError: 322616:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12669.956738] LustreError: 322616:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12675.041729] LustreError: 322615:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12678.051114] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12678.600820] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12685.289312] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [12697.584977] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12707.172424] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12717.549874] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12718.187443] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12726.805613] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [12741.089911] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12765.111854] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.20@tcp (stopping) [12765.664047] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12765.675322] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12765.678336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12765.701283] Lustre: Skipped 1 previous similar message [12769.614057] Lustre: server umount lustre-MDT0000 complete [12770.189104] LustreError: 322610:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12774.694770] LustreError: 322597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785946475 with bad export cookie 6643556222458131575 [12774.698168] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12774.709355] LustreError: 322597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12775.290821] LustreError: 322611:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12775.315330] LustreError: 322611:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12775.350289] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.20@tcp (stopping) [12775.364894] Lustre: Skipped 2 previous similar messages [12775.908748] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12775.938233] Lustre: Skipped 1 previous similar message [12779.936025] LustreError: 322610:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12779.936116] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12779.959305] LustreError: 322610:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12779.978117] Lustre: Skipped 1 previous similar message [12781.455966] Lustre: server umount lustre-MDT0001 complete [12789.656643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [12801.641050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12821.044476] LustreError: 324453:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12821.119211] LustreError: 324453:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12821.308229] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12821.463590] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [12828.679654] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [12834.800283] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12839.333787] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12839.802441] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12845.040534] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12845.940489] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [12854.793655] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12857.167710] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12866.039375] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12868.357791] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [12875.758533] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12875.778488] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12875.825847] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12875.853775] Lustre: Skipped 1 previous similar message [12877.796638] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12877.799564] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12885.988107] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12886.002426] Lustre: Skipped 4 previous similar messages [12890.079110] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12890.515633] Lustre: server umount lustre-OST0000 complete [12896.226332] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12896.249242] Lustre: Skipped 1 previous similar message [12901.351963] LustreError: 326248:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12901.379444] LustreError: 326248:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [12901.680133] Lustre: server umount lustre-MDT0000 complete [12906.979491] LustreError: 322596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785946607 with bad export cookie 6643556222458139387 [12906.990215] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12906.994322] LustreError: 322596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12907.663801] Lustre: server umount lustre-MDT0001 complete [12921.265542] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [12925.950906] Key type lgssc unregistered [12926.369278] LNet: 328924:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12926.382658] LNetError: 328924:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12926.401755] LNet: Removed LNI 192.168.203.120@tcp [12927.194180] Key type .llcrypt unregistered [12927.195857] Key type ._llcrypt unregistered [12960.526394] Key type ._llcrypt registered [12960.531667] Key type .llcrypt registered [12960.714028] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [12962.739985] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12962.768307] alg: No test for adler32 (adler32-zlib) [12964.052956] Lustre: Lustre: Build Version: 2.17.56_50_g82e980f [12964.464464] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [12966.249295] Key type lgssc registered [12967.737539] Lustre: Echo OBD driver; http://www.lustre.org/ [12976.506236] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12976.532943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12978.363711] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12983.607103] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [12987.872875] LustreError: 330343:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12987.892545] LustreError: 330343:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12992.997148] LustreError: 330344:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12993.187769] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12993.688148] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12999.519994] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [13008.546533] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13015.154582] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13019.107415] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13019.116064] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13019.143434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13023.713417] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13023.721661] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13023.756171] Lustre: Skipped 1 previous similar message [13027.481862] Lustre: server umount lustre-MDT0000 complete [13028.837228] LustreError: 330339:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13028.872810] LustreError: 330339:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13033.954135] LustreError: 330343:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13034.004988] LustreError: 330343:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13035.492889] LustreError: 330323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785946736 with bad export cookie 7662050506037518733 [13035.504330] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13035.506061] LustreError: 330323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13036.393693] Lustre: server umount lustre-MDT0001 complete [13061.006213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13061.472169] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [13061.550898] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13061.922262] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13070.654280] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [13083.011930] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13083.202598] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [13083.262702] Lustre: 332932:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13083.772646] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13090.144029] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [13100.210720] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13108.200907] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13122.059007] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13122.400731] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [13122.407798] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [13122.420420] LustreError: 334821:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [13122.434341] LustreError: 334821:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [13122.442917] LustreError: 334821:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [13122.542222] Lustre: server umount lustre-OST0000 complete [13122.545181] LustreError: 334821:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [13135.729640] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13136.007868] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [13136.272330] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13143.364356] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [13145.932660] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [13156.370096] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13166.203078] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13166.331319] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [13166.477088] Lustre: lustre-OST0001: new disk, initializing [13166.482221] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [13166.500544] Lustre: 336366:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13166.622200] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13173.891191] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13173.911975] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13174.028438] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13175.285966] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [13187.793832] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13194.720325] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13194.730645] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13194.742342] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [13194.747353] Lustre: Skipped 1 previous similar message [13194.770834] Lustre: Skipped 1 previous similar message [13197.953166] Lustre: server umount lustre-OST0001 complete [13199.843439] LustreError: 335289:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13199.873720] LustreError: 335289:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13204.965960] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13204.982581] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13204.984989] LustreError: Skipped 1 previous similar message [13205.029940] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13205.038854] Lustre: Skipped 1 previous similar message [13209.344478] Lustre: server umount lustre-OST0000 complete [13215.203610] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13215.218267] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13215.221730] Lustre: Skipped 2 previous similar messages [13215.242103] Lustre: Skipped 1 previous similar message [13221.302191] Lustre: server umount lustre-MDT0000 complete [13224.418481] LustreError: 332945:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13224.436154] LustreError: 332945:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [13226.684342] LustreError: 332926:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785946927 with bad export cookie 7662050506037519286 [13226.688706] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13226.696689] LustreError: 332926:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13227.464386] Lustre: server umount lustre-MDT0001 complete [13241.393913] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [13245.433888] Key type lgssc unregistered [13245.921782] LNet: 338327:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13245.931306] LNetError: 338327:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13245.957354] LNet: Removed LNI 192.168.203.120@tcp [13247.321765] Key type .llcrypt unregistered [13247.323222] Key type ._llcrypt unregistered [13355.704833] Key type ._llcrypt registered [13355.713051] Key type .llcrypt registered [13355.904455] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [13379.913107] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [13383.021080] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13383.142899] alg: No test for adler32 (adler32-zlib) [13384.703073] Lustre: Lustre: Build Version: 2.17.56_50_g82e980f [13385.308042] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [13387.213938] Key type lgssc registered [13389.995490] Lustre: Echo OBD driver; http://www.lustre.org/ [13445.989917] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13446.025323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13447.287492] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13447.317249] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13447.392408] Lustre: lustre-MDT0000: new disk, initializing [13447.459224] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13447.487990] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13453.211385] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [13466.064777] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13466.200554] Lustre: 343278:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13466.300155] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13466.310400] Lustre: Skipped 1 previous similar message [13466.446462] Lustre: lustre-MDT0001: new disk, initializing [13466.605710] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13466.654934] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13466.674120] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13474.820737] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [13492.915554] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13500.247572] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13510.550227] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13510.954041] Lustre: lustre-OST0000: new disk, initializing [13510.957839] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13510.975465] Lustre: 345249:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13511.079452] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13517.471101] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13517.492656] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13517.696344] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13520.546094] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [13533.488789] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13539.440475] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [13539.708540] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13547.508996] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [13547.835632] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13553.642889] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13553.643762] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13553.669738] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13553.716180] Lustre: Skipped 1 previous similar message [13557.596691] Lustre: server umount lustre-OST0000 complete [13563.875056] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13563.888475] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13563.890338] Lustre: Skipped 1 previous similar message [13563.915584] Lustre: Skipped 2 previous similar messages [13568.992410] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13569.004892] Lustre: Skipped 1 previous similar message [13574.116092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13577.183141] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13577.525566] Lustre: server umount lustre-MDT0000 complete [13582.580590] LustreError: 343270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785947283 with bad export cookie 15377295585962981943 [13582.591773] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13582.601656] LustreError: 343270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13583.081697] Lustre: server umount lustre-MDT0001 complete [13595.772299] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 12:28:13 (1785947293) [13608.793561] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 12:28:26 (1785947306) [13619.801725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13620.448961] LustreError: 348007:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13620.473372] LustreError: 348007:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [13620.576860] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13625.829583] LustreError: 348008:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13626.544207] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [13630.945665] LustreError: 348007:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13636.070823] LustreError: 348008:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13636.979085] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13637.669017] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13645.495740] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [13658.301866] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13667.041439] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13676.598958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13677.067988] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13685.060353] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [13696.310822] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13709.657553] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [13712.247129] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [13715.512415] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [13717.103980] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [13719.287202] Lustre: Failing over lustre-MDT0000 [13719.519716] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13719.530033] LustreError: Skipped 1 previous similar message [13719.540161] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13719.574633] LustreError: 348007:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13721.807552] Lustre: server umount lustre-MDT0000 complete [13722.107917] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13722.136996] Lustre: Skipped 1 previous similar message [13729.152291] LustreError: 348004:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13729.187377] LustreError: 348004:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [13737.443899] Lustre: 341497:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785947422/real 1785947422] req@00000000d4d29546 x1872701250686720/t0(0) o400->MGC192.168.203.120@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1785947438 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13737.486747] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13745.308818] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13745.533862] LustreError: 348002:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13745.565747] LustreError: 348002:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [13747.551113] LustreError: 351231:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [13747.571253] LustreError: 351231:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@00000000019936ea x1872701250690432/t0(0) o250->MGC192.168.203.120@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1785947448 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13747.626186] LustreError: 351231:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [13747.708624] LustreError: 341493:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000012363bb4 x1872701250692096/t0(0) o250->MGC192.168.203.120@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13748.276434] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13748.380113] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [13750.663980] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [13753.340554] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [13753.381751] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [13753.463790] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [13754.729817] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [13769.358085] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13772.457098] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [13779.110079] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [13782.889639] Lustre: Failing over lustre-MDT0000 [13783.628623] Lustre: server umount lustre-MDT0000 complete [13784.034609] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13784.047543] Lustre: Skipped 1 previous similar message [13784.049087] LustreError: 348737:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13784.061100] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13784.110820] LustreError: 348737:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [13800.415090] Lustre: 341496:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785947484/real 1785947484] req@00000000d7c71e9d x1872701250711040/t0(0) o400->MGC192.168.203.120@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1785947500 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13800.460893] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13805.370445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13810.659517] LustreError: 341493:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000000be6a5e6 x1872701250716672/t0(0) o250->MGC192.168.203.120@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13811.101728] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13811.179891] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [13813.113157] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [13816.310156] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [13816.319158] Lustre: Skipped 2 previous similar messages [13816.366677] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [13816.437775] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [13816.634420] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [13830.009140] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13832.503461] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [13841.890468] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13841.902277] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13841.904941] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13841.922536] Lustre: Skipped 3 previous similar messages [13841.935710] Lustre: Skipped 3 previous similar messages [13847.625428] Lustre: server umount lustre-MDT0000 complete [13852.130350] LustreError: 348008:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13852.162764] LustreError: 348008:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 26 previous similar messages [13852.302715] LustreError: 347988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785947552 with bad export cookie 15377295585962984652 [13852.314468] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13852.326732] LustreError: 347988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13853.586583] Lustre: server umount lustre-MDT0001 complete [13869.429667] Lustre: server umount lustre-OST0000 complete [13889.046911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13889.834867] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13895.235507] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [13906.119650] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13912.223686] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [13921.255558] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13928.385047] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13937.080576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13947.097993] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [13955.984411] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13960.194995] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [13970.401518] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13970.419834] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13970.430540] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13970.461723] Lustre: Skipped 1 previous similar message [13970.497912] Lustre: Skipped 4 previous similar messages [13976.462763] Lustre: server umount lustre-OST0000 complete [13983.204312] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13988.526423] Lustre: server umount lustre-MDT0000 complete [13990.888684] LustreError: 354381:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13990.919214] LustreError: 354381:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [13993.451025] LustreError: 354362:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785947694 with bad export cookie 15377295585962985394 [13993.458863] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13993.999573] Lustre: server umount lustre-MDT0001 complete [14009.436496] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [14012.829420] Key type lgssc unregistered [14013.550709] LNet: 357941:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14013.563704] LNetError: 357941:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14013.587230] LNet: Removed LNI 192.168.203.120@tcp [14014.636528] Key type .llcrypt unregistered [14014.638356] Key type ._llcrypt unregistered [14050.806265] Key type ._llcrypt registered [14050.809367] Key type .llcrypt registered [14050.913431] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [14052.642388] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14052.670702] alg: No test for adler32 (adler32-zlib) [14054.013269] Lustre: Lustre: Build Version: 2.17.56_50_g82e980f [14054.257604] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [14055.943832] Key type lgssc registered [14058.041280] Lustre: Echo OBD driver; http://www.lustre.org/ [14069.850755] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14069.882185] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14071.686190] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14079.725352] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [14082.027075] LustreError: 359461:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14082.074483] LustreError: 359461:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [14087.138612] LustreError: 359462:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14092.259750] LustreError: 359461:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14092.665993] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14093.072349] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14099.657199] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [14108.767310] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14115.708369] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14124.729887] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14125.113903] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14134.277232] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [14141.489727] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [14144.110866] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14156.782443] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14156.791747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14156.807473] Lustre: Skipped 2 previous similar messages [14156.830571] Lustre: Skipped 2 previous similar messages [14161.896708] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14161.915452] Lustre: Skipped 2 previous similar messages [14162.601502] Lustre: server umount lustre-MDT0000 complete [14167.013623] LustreError: 361305:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14167.035660] LustreError: 361305:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [14167.288693] LustreError: 359441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785947867 with bad export cookie 3641705102035852584 [14167.297322] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14167.303271] LustreError: 359441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14167.896339] Lustre: server umount lustre-MDT0001 complete [14183.757963] Lustre: server umount lustre-OST0000 complete [14206.185316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14206.936807] LustreError: 363180:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14207.114630] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14215.353694] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [14216.743490] LustreError: 363181:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14216.751872] LustreError: 363181:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [14226.987454] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14227.521392] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14232.653739] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [14242.914946] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14250.590451] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14259.485076] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14260.028570] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14269.312168] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [14277.420135] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [14279.687337] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14298.080431] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14298.101449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14298.117197] Lustre: Skipped 1 previous similar message [14298.134057] Lustre: Skipped 1 previous similar message [14301.408045] Lustre: server umount lustre-MDT0000 complete [14303.218505] LustreError: 363909:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14303.252131] LustreError: 363909:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [14307.663169] LustreError: 365025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785948008 with bad export cookie 3641705102035853697 [14307.683245] LustreError: 365025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14307.692874] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14308.487768] Lustre: server umount lustre-MDT0001 complete [14329.032210] Lustre: server umount lustre-OST0000 complete [14351.114610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14352.077526] LustreError: 367098:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14352.303470] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14359.157977] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [14371.185889] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14379.269819] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [14392.282818] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14400.473886] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14410.958898] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14411.457346] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14411.466912] Lustre: Skipped 1 previous similar message [14420.514206] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [14421.714673] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [14430.256026] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14442.465345] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14442.472609] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14442.489667] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14442.527593] Lustre: Skipped 2 previous similar messages [14447.599896] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14447.626560] Lustre: Skipped 1 previous similar message [14452.193463] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14453.257375] LustreError: 368922:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14453.286800] LustreError: 368922:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [14453.565577] Lustre: server umount lustre-OST0000 complete [14463.468986] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14463.477009] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14463.486790] Lustre: Skipped 1 previous similar message [14463.498974] Lustre: Skipped 2 previous similar messages [14469.851232] Lustre: server umount lustre-MDT0000 complete [14476.346789] LustreError: 367080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785948176 with bad export cookie 3641705102035854719 [14476.375994] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14477.454556] Lustre: server umount lustre-MDT0001 complete [14494.530238] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [14501.125762] Key type lgssc unregistered [14502.009141] LNet: 370642:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14502.056462] LNetError: 370642:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14502.090118] LNet: Removed LNI 192.168.203.120@tcp [14504.247462] Key type .llcrypt unregistered [14504.255479] Key type ._llcrypt unregistered [14545.233338] Key type ._llcrypt registered [14545.234645] Key type .llcrypt registered [14545.413411] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [14547.146084] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14547.256490] alg: No test for adler32 (adler32-zlib) [14549.288404] Lustre: Lustre: Build Version: 2.17.56_50_g82e980f [14549.806695] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [14551.536528] Key type lgssc registered [14553.042448] Lustre: Echo OBD driver; http://www.lustre.org/ [14563.399688] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14563.437168] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14563.447935] LustreError: 372085:0:(osd_handler.c:8554:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [14563.485714] LustreError: 372085:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [14563.494352] LustreError: 372085:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [14563.502242] LustreError: 372085:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [14563.510615] LustreError: 372085:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [14572.700284] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [14572.710400] LustreError: 372482:0:(osd_handler.c:8554:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [14572.731557] LustreError: 372482:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [14572.733939] LustreError: 372482:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [14572.738801] LustreError: 372482:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [14572.743958] LustreError: 372482:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [14597.020223] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [14600.866580] Key type lgssc unregistered [14601.328484] LNet: 373491:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14601.337542] LNetError: 373491:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14601.363556] LNet: Removed LNI 192.168.203.120@tcp [14602.681897] Key type .llcrypt unregistered [14602.685969] Key type ._llcrypt unregistered [14626.760421] Key type ._llcrypt registered [14626.762193] Key type .llcrypt registered [14626.873275] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [14646.173813] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [14648.324282] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14648.354650] alg: No test for adler32 (adler32-zlib) [14649.635794] Lustre: Lustre: Build Version: 2.17.56_50_g82e980f [14650.059154] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [14651.823126] Key type lgssc registered [14653.645474] Lustre: Echo OBD driver; http://www.lustre.org/ [14707.745858] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14707.772083] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14709.332566] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14709.424521] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14709.599512] Lustre: lustre-MDT0000: new disk, initializing [14709.766200] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14709.808306] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14715.972241] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [14729.364163] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14729.520731] Lustre: 376657:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14729.576041] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14729.578643] Lustre: Skipped 1 previous similar message [14729.652785] Lustre: lustre-MDT0001: new disk, initializing [14729.766515] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14729.822416] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14729.845266] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14735.844636] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [14748.134732] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14755.686462] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14765.053447] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14765.454484] Lustre: lustre-OST0000: new disk, initializing [14765.461503] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14765.469159] Lustre: 378626:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14765.606372] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14770.757662] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14770.769470] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14770.916709] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14773.207408] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [14784.639984] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14790.316769] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [14790.550218] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [14796.136602] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [14796.443299] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [14801.888121] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14801.902489] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14801.904882] LustreError: Skipped 1 previous similar message [14801.907180] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14801.944905] Lustre: Skipped 1 previous similar message [14805.536118] Lustre: server umount lustre-OST0000 complete [14811.617573] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14811.626042] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14811.638038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14811.642206] Lustre: Skipped 1 previous similar message [14816.397046] Lustre: server umount lustre-MDT0000 complete [14821.770846] LustreError: 376648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785948522 with bad export cookie 6915314926988243190 [14821.772092] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14821.786051] LustreError: 376648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14822.814646] Lustre: server umount lustre-MDT0001 complete [14833.359755] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [14835.841547] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 12:48:54 (1785948534) [14845.492130] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14846.579855] LustreError: 381087:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14846.614725] LustreError: 381087:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [14846.748374] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14852.065990] LustreError: 381088:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14855.131325] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [14857.185540] LustreError: 381087:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14862.305970] LustreError: 381088:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14867.423863] LustreError: 381087:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14867.799147] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14868.127672] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14873.713713] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [14884.802913] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14892.948425] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14902.137100] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14902.665973] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14910.744769] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [14920.010832] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14930.503783] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14930.802446] Lustre: lustre-OST0001: new disk, initializing [14930.807110] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [14930.813149] Lustre: 383928:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14930.958342] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [14937.899643] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [14937.911681] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [14938.080369] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [14942.569673] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [14957.042808] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14963.717501] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14963.747224] Lustre: Skipped 1 previous similar message [14963.758918] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14963.777518] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [14963.809551] Lustre: Skipped 3 previous similar messages [14968.365907] Lustre: server umount lustre-OST0001 complete [14968.812752] LustreError: 382915:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14968.850580] LustreError: 382915:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [14989.360654] LustreError: 382915:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14989.406143] LustreError: 382915:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [14993.891423] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14993.918822] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14993.969719] Lustre: Skipped 1 previous similar message [14993.977952] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14994.003257] Lustre: Skipped 1 previous similar message [14997.810744] Lustre: server umount lustre-OST0000 complete [15004.035462] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15004.044628] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15004.048110] Lustre: Skipped 2 previous similar messages [15004.070087] Lustre: Skipped 3 previous similar messages [15010.533330] Lustre: server umount lustre-MDT0000 complete [15016.509616] LustreError: 381069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785948717 with bad export cookie 6915314926988244072 [15016.519445] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15016.524672] LustreError: 381069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15023.183635] Lustre: server umount lustre-MDT0001 complete [15038.845114] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [15044.965247] Key type lgssc unregistered [15045.664428] LNet: 385926:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15045.683972] LNetError: 385926:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15045.740122] LNet: Removed LNI 192.168.203.120@tcp [15048.036323] Key type .llcrypt unregistered [15048.041552] Key type ._llcrypt unregistered [15124.083300] Key type ._llcrypt registered [15124.091323] Key type .llcrypt registered [15124.309431] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [15126.766465] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15126.807484] alg: No test for adler32 (adler32-zlib) [15128.281321] Lustre: Lustre: Build Version: 2.17.56_50_g82e980f [15128.663751] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [15130.431159] Key type lgssc registered [15132.023987] Lustre: Echo OBD driver; http://www.lustre.org/ [15143.013702] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15143.056318] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15144.490868] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [15144.522610] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15144.684990] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15152.071938] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [15168.697590] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15169.014463] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [15169.125552] Lustre: 388734:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15169.827972] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15177.707099] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [15189.524385] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15198.125187] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15203.297290] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15203.298103] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15203.320183] Lustre: Skipped 1 previous similar message [15207.511906] Lustre: server umount lustre-MDT0000 complete [15212.928343] LustreError: 388726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785948913 with bad export cookie 7689250167457958194 [15212.936849] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15212.961391] LustreError: 388726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15213.357304] Lustre: server umount lustre-MDT0001 complete [15228.062414] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 12:55:26 (1785948926) [15237.131344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15237.780939] LustreError: 391252:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15237.808472] LustreError: 391252:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [15237.891244] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15243.232143] LustreError: 391253:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15244.585022] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [15248.361276] LustreError: 391252:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15253.476865] LustreError: 391253:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15255.139552] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15255.653496] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15261.351352] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [15271.886717] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15278.873737] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15287.107700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15287.532989] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [15287.860133] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15296.975970] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [15308.586708] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15318.108471] Lustre: Permanently deactivating lustre-OST0000 [15318.135974] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [15324.227509] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [15327.715617] LustreError: 394196:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [15327.742295] LustreError: 394196:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [15327.758328] LustreError: 394196:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [15333.865041] LustreError: 394244:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [15333.913112] LustreError: 394244:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [15352.287800] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15352.844285] Lustre: server umount lustre-OST0000 complete [15358.325351] LustreError: 394496:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [15358.333935] LustreError: 394496:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [15362.020116] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15362.064192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15362.071779] Lustre: Skipped 2 previous similar messages [15367.140936] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15367.158653] Lustre: Skipped 1 previous similar message [15372.255992] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15372.538480] Lustre: server umount lustre-MDT0000 complete [15377.390026] LustreError: 392177:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15377.414479] LustreError: 392177:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [15377.676132] LustreError: 391234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785949078 with bad export cookie 7689250167457958775 [15377.695520] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15388.490567] Lustre: server umount lustre-MDT0001 complete [15398.708761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15404.795665] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [15407.683893] LustreError: 395656:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [15414.868737] Lustre: 395996:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.120@tcp [15417.753582] Lustre: server umount MGS complete [15431.546514] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15432.009950] LustreError: 396720:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15432.028694] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [15432.040467] Lustre: Skipped 1 previous similar message [15432.151301] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15438.461676] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [15449.602927] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15450.034941] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [15450.180471] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15455.657956] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [15465.222804] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15471.905576] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15473.746919] Lustre: Permanently reactivating lustre-OST0000 [15473.766093] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [15473.768750] Lustre: Skipped 2 previous similar messages [15502.819455] LustreError: 398503:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [15511.860533] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15512.376101] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15520.559295] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [15529.848335] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15538.151105] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15538.172912] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15538.173548] Lustre: Skipped 1 previous similar message [15538.174094] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15538.174098] Lustre: Skipped 1 previous similar message [15548.388522] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15548.402840] Lustre: Skipped 4 previous similar messages [15551.967278] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15552.231478] Lustre: server umount lustre-OST0000 complete [15557.603617] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15557.629270] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15557.651354] Lustre: Skipped 1 previous similar message [15568.867412] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15568.881177] Lustre: Skipped 6 previous similar messages [15570.399139] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15570.712833] Lustre: server umount lustre-MDT0000 complete [15573.986639] LustreError: 396716:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15574.003643] LustreError: 396716:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [15576.712663] LustreError: 398297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785949277 with bad export cookie 7689250167457959825 [15576.713951] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15576.721300] LustreError: 398297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [15577.579916] Lustre: server umount lustre-MDT0001 complete [15597.169122] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [15602.903959] Key type lgssc unregistered [15603.365252] LNet: 401078:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15603.380551] LNetError: 401078:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15603.404135] LNet: Removed LNI 192.168.203.120@tcp [15605.150564] Key type .llcrypt unregistered [15605.152347] Key type ._llcrypt unregistered [15630.803140] Key type ._llcrypt registered [15630.806492] Key type .llcrypt registered [15630.958158] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [15649.564842] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [15650.908931] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15650.982891] alg: No test for adler32 (adler32-zlib) [15652.618852] Lustre: Lustre: Build Version: 2.17.56_50_g82e980f [15653.154433] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [15654.855167] Key type lgssc registered [15656.406492] Lustre: Echo OBD driver; http://www.lustre.org/ [15698.994906] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15699.046736] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15700.438921] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15700.555326] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15700.682106] Lustre: lustre-MDT0000: new disk, initializing [15700.802231] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15700.844401] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15707.672252] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [15720.871063] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15721.033956] Lustre: 404241:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15721.125814] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15721.132441] Lustre: Skipped 1 previous similar message [15721.298944] Lustre: lustre-MDT0001: new disk, initializing [15721.442379] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15721.529107] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15721.557688] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15728.246886] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [15738.416560] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15746.409552] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15754.624560] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15755.076265] Lustre: lustre-OST0000: new disk, initializing [15755.080278] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15755.086590] Lustre: 406210:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15755.172091] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15760.446100] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15760.454450] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15760.570138] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15762.481178] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [15773.367744] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15779.355597] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15779.850342] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15785.998997] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15786.238782] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15791.592039] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15791.598704] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15791.600394] LustreError: Skipped 1 previous similar message [15791.619156] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15791.621333] Lustre: Skipped 1 previous similar message [15791.645022] Lustre: Skipped 1 previous similar message [15795.137921] Lustre: server umount lustre-OST0000 complete [15801.829553] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15801.852611] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15801.862972] Lustre: Skipped 1 previous similar message [15806.514132] Lustre: server umount lustre-MDT0000 complete [15811.271043] LustreError: 405045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785949511 with bad export cookie 6008613336138278529 [15811.277114] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15811.290125] LustreError: 405045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15811.921478] Lustre: server umount lustre-MDT0001 complete [15822.412919] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [15824.406120] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 13:05:23 (1785949523) [15833.209561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15833.733612] LustreError: 408671:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15833.759238] LustreError: 408671:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [15833.859165] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15839.201994] LustreError: 408672:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15841.308247] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [15844.331942] LustreError: 408672:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15850.471107] LustreError: 408671:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15850.504789] LustreError: 408671:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [15855.585206] LustreError: 408672:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15857.049231] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15857.425136] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15864.187602] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [15874.217374] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15881.597064] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15893.813448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15894.345341] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15903.935990] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [15915.237034] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15918.054223] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [15920.931890] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [15930.340546] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15930.352164] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15930.361060] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15930.381551] Lustre: Skipped 2 previous similar messages [15934.418526] Lustre: server umount lustre-OST0000 complete [15939.556304] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15939.572749] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15939.610725] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15939.632956] Lustre: Skipped 1 previous similar message [15945.699513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15945.718623] Lustre: Skipped 3 previous similar messages [15946.194211] Lustre: server umount lustre-MDT0000 complete [15950.816311] LustreError: 409401:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15950.842413] LustreError: 409401:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [15952.558674] LustreError: 408651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785949653 with bad export cookie 6008613336138279411 [15952.569830] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15952.863487] Lustre: server umount lustre-MDT0001 complete [15969.961705] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [15974.445657] Key type lgssc unregistered [15974.810105] LNet: 412403:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15974.818744] LNetError: 412403:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15975.848773] LNet: Removed LNI 192.168.203.120@tcp [15977.207527] Key type .llcrypt unregistered [15977.210360] Key type ._llcrypt unregistered [16012.197826] Key type ._llcrypt registered [16012.202856] Key type .llcrypt registered [16012.489605] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [16013.367951] Key type .llcrypt unregistered [16013.385692] Key type ._llcrypt unregistered [16036.397938] Key type ._llcrypt registered [16036.400584] Key type .llcrypt registered [16036.480722] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [16037.752824] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16037.885224] alg: No test for adler32 (adler32-zlib) [16039.083439] Lustre: Lustre: Build Version: 2.17.56_50_g82e980f [16039.541247] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [16041.312488] Key type lgssc registered [16042.651357] Lustre: Echo OBD driver; http://www.lustre.org/ [16053.759758] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16053.783638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16055.559261] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16062.924296] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [16066.026503] LustreError: 414626:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16066.050313] LustreError: 414626:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [16071.139871] LustreError: 414627:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16075.827706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16076.260446] LustreError: 414626:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16076.522381] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16080.640687] LustreError: 414627:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16080.663826] LustreError: 414627:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [16080.673141] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [16085.985261] LustreError: 414626:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16086.009268] LustreError: 414626:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [16086.214182] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [16096.228016] LustreError: 414627:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16096.253948] LustreError: 414627:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [16096.614763] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16105.284498] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16105.848470] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16112.195721] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [16126.436060] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16126.441146] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16126.449154] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16126.469853] Lustre: Skipped 1 previous similar message [16128.910638] Lustre: server umount lustre-OST0000 complete [16136.142529] Lustre: server umount lustre-MDT0000 complete [16136.677870] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16136.686837] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16136.708980] LustreError: 414623:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16136.711390] Lustre: Skipped 2 previous similar messages [16136.747795] LustreError: 414623:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [16141.157816] LustreError: 414606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785949841 with bad export cookie 8444089397057634104 [16141.158726] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16141.167338] LustreError: 414606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [16141.747950] Lustre: server umount lustre-MDT0001 complete [16155.474477] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [16159.704962] Key type lgssc unregistered [16160.013266] LNet: 417780:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16160.018568] LNetError: 417780:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16160.054510] LNet: Removed LNI 192.168.203.120@tcp [16161.348131] Key type .llcrypt unregistered [16161.349365] Key type ._llcrypt unregistered [16194.374626] Key type ._llcrypt registered [16194.378275] Key type .llcrypt registered [16194.519184] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [16196.125065] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16196.470117] alg: No test for adler32 (adler32-zlib) [16197.915239] Lustre: Lustre: Build Version: 2.17.56_50_g82e980f [16198.528964] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [16200.273922] Key type lgssc registered [16202.609687] Lustre: Echo OBD driver; http://www.lustre.org/ [16215.322624] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16215.337563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16230.943321] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [16236.063245] LustreError: 419171:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.120@tcp: failed processing log, type 4: rc = -110 [16262.751660] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16276.746397] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [16287.261297] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16296.311869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16296.995459] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16297.118154] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [16303.075823] LustreError: 420212:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16303.098386] LustreError: 420212:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [16303.163429] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [16308.199525] LustreError: 420210:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16308.231506] LustreError: 420210:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [16313.321374] LustreError: 420212:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16313.346162] LustreError: 420212:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [16315.033977] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16315.583613] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16320.664533] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [16330.656575] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16338.381052] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16351.216788] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16351.225944] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16351.244774] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16351.726286] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16351.754364] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16355.785946] Lustre: server umount lustre-OST0000 complete [16361.445895] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16361.456064] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16361.475459] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16367.074461] LustreError: 420215:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16367.098390] LustreError: 420215:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [16367.220856] Lustre: server umount lustre-MDT0000 complete [16371.561321] LustreError: 421995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785950072 with bad export cookie 9564649336927720497 [16371.568023] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16371.576849] LustreError: 421995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [16372.259439] Lustre: server umount lustre-MDT0001 complete [16387.195169] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [16391.478118] Key type lgssc unregistered [16391.968763] LNet: 422751:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16391.982941] LNetError: 422751:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16392.026522] LNet: Removed LNI 192.168.203.120@tcp [16393.252234] Key type .llcrypt unregistered [16393.253647] Key type ._llcrypt unregistered [16424.647871] Key type ._llcrypt registered [16424.651504] Key type .llcrypt registered [16424.879632] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [16427.201225] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16427.574356] alg: No test for adler32 (adler32-zlib) [16429.305569] Lustre: Lustre: Build Version: 2.17.56_50_g82e980f [16429.871857] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [16431.747630] Key type lgssc registered [16434.252634] Lustre: Echo OBD driver; http://www.lustre.org/ [16444.036426] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16444.046722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16445.635722] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16453.427806] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [16456.169788] LustreError: 424172:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16456.197501] LustreError: 424172:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [16457.185992] LustreError: 424171:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16462.324785] LustreError: 424172:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16465.908916] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16466.868994] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16473.649280] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [16482.812897] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16491.284449] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16500.377944] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16500.686124] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16510.605430] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [16517.101979] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [16519.591590] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16526.750360] LustreError: 424168:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [16527.016061] Lustre: server umount lustre-MDT0000 complete [16527.347066] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16527.382203] Lustre: Skipped 1 previous similar message [16529.979846] LustreError: 424150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.20@tcp arrived at 1785950230 with bad export cookie 9047705227006972875 [16530.005254] LustreError: 424150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [16532.458584] LustreError: 424168:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16532.495497] LustreError: 424168:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [16532.628355] LustreError: 424152:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785950233 with bad export cookie 9047705227006971979 [16532.647991] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16532.649815] LustreError: 424152:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [16537.568315] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16537.588989] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16537.617641] Lustre: Skipped 1 previous similar message [16537.630457] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16539.144299] Lustre: server umount lustre-OST0000 complete [16542.696441] LustreError: 424171:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16542.738227] LustreError: 424171:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [16547.802786] Lustre: server umount lustre-MDT0001 complete [16562.501405] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [16566.880400] Key type lgssc unregistered [16567.348666] LNet: 427863:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16567.360329] LNetError: 427863:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16567.375782] LNet: Removed LNI 192.168.203.120@tcp [16569.280734] Key type .llcrypt unregistered [16569.283500] Key type ._llcrypt unregistered [16603.526347] Key type ._llcrypt registered [16603.527542] Key type .llcrypt registered [16603.644479] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [16606.237875] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16606.397044] alg: No test for adler32 (adler32-zlib) [16608.247994] Lustre: Lustre: Build Version: 2.17.56_50_g82e980f [16609.160961] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [16611.140085] Key type lgssc registered [16614.055633] Lustre: Echo OBD driver; http://www.lustre.org/ [16627.604695] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16627.627201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16629.456087] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16635.998885] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [16638.955347] LustreError: 429282:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16638.978330] LustreError: 429282:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [16644.090157] LustreError: 429283:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16647.068399] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16647.863687] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16653.860154] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [16665.384300] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16677.635155] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16691.094159] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16691.575432] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16698.695839] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [16698.808180] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [16707.672902] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16714.211745] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16714.229575] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [16714.238228] Lustre: Skipped 1 previous similar message [16714.640693] LustreError: 429278:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [16714.787965] Lustre: server umount lustre-MDT0001 complete [16719.329919] LustreError: 431677:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16719.358513] LustreError: 431677:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [16724.451253] LustreError: 429282:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16724.453133] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16724.500394] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16724.532554] Lustre: Skipped 1 previous similar message [16724.552963] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16729.493845] Lustre: server umount lustre-OST0000 complete [16734.720333] LustreError: 429282:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16734.753955] LustreError: 429282:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [16737.395837] Lustre: server umount lustre-MDT0000 complete [16767.259127] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [16771.984294] Key type lgssc unregistered [16772.594191] LNet: 432996:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16772.607993] LNetError: 432996:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16772.648628] LNet: Removed LNI 192.168.203.120@tcp [16774.285716] Key type .llcrypt unregistered [16774.288033] Key type ._llcrypt unregistered [16817.311484] Key type ._llcrypt registered [16817.312965] Key type .llcrypt registered [16817.440947] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [16838.700432] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [16841.204381] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16841.280573] alg: No test for adler32 (adler32-zlib) [16842.566382] Lustre: Lustre: Build Version: 2.17.56_50_g82e980f [16843.040895] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [16844.776261] Key type lgssc registered [16846.605676] Lustre: Echo OBD driver; http://www.lustre.org/ [16895.771441] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [16916.056028] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16916.095160] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16917.456423] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16917.498270] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16917.557186] Lustre: lustre-MDT0000: new disk, initializing [16917.613772] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16917.625498] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16922.850531] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [16941.389875] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16941.631077] Lustre: lustre-OST0000: new disk, initializing [16941.636386] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16941.644369] Lustre: Skipped 1 previous similar message [16941.655942] Lustre: 438041:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16941.752150] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16950.674818] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [16951.440708] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [16951.468483] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [16951.506748] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [16965.082796] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16984.509121] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16984.733301] Lustre: 437020:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16984.839892] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16984.968441] Lustre: lustre-MDT0001: new disk, initializing [16985.079828] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16985.136830] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [16985.170944] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [16993.381929] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [17007.150978] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [17007.495574] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [17014.186146] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [17014.478865] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [17024.942300] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.20@tcp (stopping) [17025.506675] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17025.510458] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [17025.535794] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [17025.565297] Lustre: Skipped 1 previous similar message [17027.255531] Lustre: server umount lustre-MDT0001 complete [17035.126841] LustreError: 437028:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [17035.141871] LustreError: 437028:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [17035.751301] LustreError: 437031:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17035.770947] LustreError: 437031:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [17040.865791] LustreError: 437032:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17040.893676] LustreError: 437032:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [17045.994624] LustreError: 437028:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17046.014262] LustreError: 437028:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [17051.105857] LustreError: 437031:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17051.107807] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17051.144749] Lustre: Skipped 1 previous similar message [17051.154437] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17056.245534] Lustre: server umount lustre-OST0000 complete [17061.348700] LustreError: 437032:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17061.383786] LustreError: 437032:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [17068.304527] Lustre: server umount lustre-MDT0000 complete [17082.820269] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [17086.746775] Key type lgssc unregistered [17087.197228] LNet: 441798:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [17087.215591] LNetError: 441798:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [17087.235776] LNet: Removed LNI 192.168.203.120@tcp [17088.480892] Key type .llcrypt unregistered [17088.484235] Key type ._llcrypt unregistered [17146.695291] Key type ._llcrypt registered [17146.699917] Key type .llcrypt registered [17146.872913] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [17147.501473] Key type .llcrypt unregistered [17147.506721] Key type ._llcrypt unregistered [17172.298349] Key type ._llcrypt registered [17172.308446] Key type .llcrypt registered [17172.556878] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [17174.196477] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [17174.348737] alg: No test for adler32 (adler32-zlib) [17175.814886] Lustre: Lustre: Build Version: 2.17.56_50_g82e980f [17176.281034] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [17178.128357] Key type lgssc registered [17180.255468] Lustre: Echo OBD driver; http://www.lustre.org/ [17215.013101] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [17215.062104] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17216.433324] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [17216.469429] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [17216.604942] Lustre: lustre-MDT0000: new disk, initializing [17216.776288] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [17216.794936] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [17223.805558] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [17237.222142] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17237.422866] Lustre: 445184:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [17237.521124] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [17237.528102] Lustre: Skipped 1 previous similar message [17237.719707] Lustre: lustre-MDT0001: new disk, initializing [17237.926278] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [17237.999362] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [17238.018908] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [17244.287947] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [17255.300159] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [17263.980548] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [17272.404819] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [17272.678654] Lustre: lustre-OST0000: new disk, initializing [17272.682566] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [17272.692745] Lustre: 447152:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [17272.813941] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [17279.582975] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [17279.594809] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [17279.780138] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [17281.059649] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [17292.757574] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [17304.544773] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [17304.556997] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17304.595482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17305.569231] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17305.579956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17305.597398] Lustre: Skipped 1 previous similar message [17305.626827] Lustre: Skipped 2 previous similar messages [17309.283628] Lustre: server umount lustre-MDT0000 complete [17314.931877] LustreError: 445179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785951015 with bad export cookie 1053391791363645640 [17314.938702] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [17314.952441] LustreError: 445179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [17315.715040] Lustre: server umount lustre-MDT0001 complete [17333.986561] Lustre: server umount lustre-OST0000 complete [17347.399520] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 13:30:46 (1785951046) [17359.387431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17359.886651] LustreError: 449185:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17359.902422] LustreError: 449185:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [17359.975613] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [17365.477107] LustreError: 449186:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17366.228796] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [17369.579922] LustreError: 449185:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17374.696811] LustreError: 449186:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17378.987975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17379.716135] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [17385.687122] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [17398.356218] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [17407.758173] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [17429.288189] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [17429.674760] Lustre: Found index 0 for lustre-OST0000, updating log [17429.999922] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [17435.182264] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [17444.935636] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [17460.287344] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [17470.953898] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [17470.967108] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17471.010097] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17475.757570] Lustre: server umount lustre-OST0000 complete [17491.812177] LNet: 1 local NIs in recovery (showing 1): 192.168.203.120@tcp [17497.574046] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [17497.584972] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17497.603534] Lustre: Skipped 1 previous similar message [17497.613936] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17497.620604] Lustre: Skipped 3 previous similar messages [17501.917639] Lustre: server umount lustre-MDT0000 complete [17504.754358] LustreError: 449180:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17504.776829] LustreError: 449180:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [17507.050539] LustreError: 449167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785951207 with bad export cookie 1053391791363647033 [17507.065065] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [17507.940884] Lustre: server umount lustre-MDT0001 complete [17519.085187] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 13:33:37 (1785951217) [17555.295257] LNet: 1 local NIs in recovery (showing 1): 192.168.203.120@tcp [18162.692449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [18163.099671] LustreError: 455606:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [18163.187965] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [18168.305990] LustreError: 455607:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [18169.580956] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [18173.417766] LustreError: 455606:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [18179.560335] LustreError: 455607:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [18179.589318] LustreError: 455607:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [18180.971997] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [18181.389419] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [18187.617214] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [18197.377364] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [18204.695340] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [18221.407229] LNet: 1 local NIs in recovery (showing 1): 192.168.203.120@tcp [18227.753021] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [18228.361823] Lustre: Found index 0 for lustre-OST0000, updating log [18228.784155] LustreError: 457443:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [18229.091478] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [18239.242229] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [18250.917545] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [18254.382175] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [18254.415928] Lustre: Skipped 1 previous similar message [18254.431808] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [18254.435724] Lustre: Skipped 4 previous similar messages [18259.786753] Lustre: server umount lustre-OST0000 complete [18265.573825] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [18265.578777] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [18265.599479] Lustre: Skipped 1 previous similar message [18265.605647] Lustre: Skipped 1 previous similar message [18266.662608] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [18268.706324] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [18268.714289] Lustre: Skipped 4 previous similar messages [18271.051783] Lustre: server umount lustre-MDT0000 complete [18271.785709] LustreError: 455603:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [18271.806063] LustreError: 455603:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [18275.379054] LustreError: 456344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785951975 with bad export cookie 1053391791363648118 [18275.386547] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [18275.395657] LustreError: 456344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [18276.324763] Lustre: server umount lustre-MDT0001 complete [18281.823127] LNet: 1 local NIs in recovery (showing 1): 192.168.203.120@tcp [18318.207530] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [18336.095040] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [18402.289883] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 13:48:20 (1785952100) [18427.624762] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [18433.705274] Key type lgssc unregistered [18434.303517] LNet: 462253:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [18434.325202] LNetError: 462253:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [18556.215050] LNet: Removed LNI 192.168.203.120@tcp [18558.233674] Key type .llcrypt unregistered [18558.236993] Key type ._llcrypt unregistered [18579.952057] Key type ._llcrypt registered [18579.954976] Key type .llcrypt registered [18580.057490] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [18581.079164] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [18581.359670] alg: No test for adler32 (adler32-zlib) [18582.833956] Lustre: Lustre: Build Version: 2.17.56_50_g82e980f [18583.171078] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [18584.871170] Key type lgssc registered [18586.744367] Lustre: Echo OBD driver; http://www.lustre.org/ [18592.741499] LNet: Added LNI 192.168.203.120@tcp100 [8/256/0/180] [18593.943618] LNet: Added LNI 192.168.203.120@tcp101 [8/256/0/180] [18595.330699] LNet: Added LNI 192.168.203.120@tcp102 [8/256/0/180] [18596.405298] LNet: Added LNI 192.168.203.120@tcp103 [8/256/0/180] [18597.450244] LNet: Added LNI 192.168.203.120@tcp104 [8/256/0/180] [18598.521940] LNet: Added LNI 192.168.203.120@tcp105 [8/256/0/180] [18599.620549] LNet: Added LNI 192.168.203.120@tcp106 [8/256/0/180] [18600.687641] LNet: Added LNI 192.168.203.120@tcp107 [8/256/0/180] [18601.893725] LNet: Added LNI 192.168.203.120@tcp108 [8/256/0/180] [18603.271929] LNet: Added LNI 192.168.203.120@tcp109 [8/256/0/180] [18604.386176] LNet: Added LNI 192.168.203.120@tcp110 [8/256/0/180] [18605.462303] LNet: Added LNI 192.168.203.120@tcp111 [8/256/0/180] [18606.523814] LNet: Added LNI 192.168.203.120@tcp112 [8/256/0/180] [18607.762793] LNet: Added LNI 192.168.203.120@tcp113 [8/256/0/180] [18608.952683] LNet: Added LNI 192.168.203.120@tcp114 [8/256/0/180] [18610.181570] LNet: Added LNI 192.168.203.120@tcp115 [8/256/0/180] [18611.205498] LNet: Added LNI 192.168.203.120@tcp116 [8/256/0/180] [18612.310240] LNet: Added LNI 192.168.203.120@tcp117 [8/256/0/180] [18613.370838] LNet: Added LNI 192.168.203.120@tcp118 [8/256/0/180] [18614.559874] LNet: Added LNI 192.168.203.120@tcp119 [8/256/0/180] [18615.546097] LNet: Added LNI 192.168.203.120@tcp120 [8/256/0/180] [18616.505957] LNet: Added LNI 192.168.203.120@tcp121 [8/256/0/180] [18617.555748] LNet: Added LNI 192.168.203.120@tcp122 [8/256/0/180] [18619.194829] LNet: Added LNI 192.168.203.120@tcp123 [8/256/0/180] [18620.169457] LNet: Added LNI 192.168.203.120@tcp124 [8/256/0/180] [18621.184427] LNet: Added LNI 192.168.203.120@tcp125 [8/256/0/180] [18622.566556] LNet: Added LNI 192.168.203.120@tcp126 [8/256/0/180] [18623.952738] LNet: Added LNI 192.168.203.120@tcp127 [8/256/0/180] [18625.060903] LNet: Added LNI 192.168.203.120@tcp128 [8/256/0/180] [18626.360817] LNet: Added LNI 192.168.203.120@tcp129 [8/256/0/180] [18627.752442] LNet: Added LNI 192.168.203.120@tcp130 [8/256/0/180] [18628.908335] LNet: Added LNI 192.168.203.120@tcp131 [8/256/0/180] [18630.287660] LNet: Added LNI 192.168.203.120@tcp132 [8/256/0/180] [18631.408380] LNet: Added LNI 192.168.203.120@tcp133 [8/256/0/180] [18632.778564] LNet: Added LNI 192.168.203.120@tcp134 [8/256/0/180] [18634.167166] LNet: Added LNI 192.168.203.120@tcp135 [8/256/0/180] [18643.147845] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [18643.185254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [18644.353032] Lustre: 465290:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.120@tcp: can't add NID 192.168.203.120@tcp130: rc = -75 [18644.480327] LNetError: 462746:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.120 [18644.490274] LNetError: Protocol error connecting to 192.168.203.120@tcp115 at host 192.168.203.120:988: is it running a compatible version of Lustre? [18644.515469] LNetError: Skipped 10 previous similar messages [18644.619961] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [18644.639073] LNet: 9 local NIs in recovery (showing 9): 192.168.203.120@tcp121, 192.168.203.120@tcp123, 192.168.203.120@tcp125, 192.168.203.120@tcp126, 192.168.203.120@tcp122, 192.168.203.120@tcp128, 192.168.203.120@tcp129, 192.168.203.120@tcp127, 192.168.203.120@tcp124 [18644.656704] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [18644.854256] Lustre: lustre-MDT0000: new disk, initializing [18645.002754] Lustre: 465344:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.120@tcp: can't add NID 192.168.203.120@tcp130: rc = -75 [18645.010263] Lustre: 465344:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [18645.060373] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [18645.076711] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [18645.536169] LNetError: 462746:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.120 [18645.558842] LNetError: 462746:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 11 previous similar messages [18645.572919] LNetError: Protocol error connecting to 192.168.203.120@tcp106 at host 192.168.203.120:988: is it running a compatible version of Lustre? [18647.788395] LNetError: 465312:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.120 [18647.807661] LNetError: 465312:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 9 previous similar messages [18647.828581] LNetError: Protocol error connecting to 192.168.203.120@tcp123 at host 192.168.203.120:988: is it running a compatible version of Lustre? [18647.851462] LNetError: Skipped 10 previous similar messages [18651.881981] LNetError: 465321:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.120 [18651.892570] LNetError: 465321:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 2 previous similar messages [18651.902909] LNetError: Protocol error connecting to 192.168.203.120@tcp123 at host 192.168.203.120:988: is it running a compatible version of Lustre? [18651.913272] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [18659.059091] LNetError: 462747:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.120 [18659.080031] LNetError: Protocol error connecting to 192.168.203.120@tcp123 at host 192.168.203.120:988: is it running a compatible version of Lustre? [18665.162181] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [18665.433277] Lustre: 465308:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [18665.553432] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [18665.556201] Lustre: Skipped 1 previous similar message [18665.695266] Lustre: lustre-MDT0001: new disk, initializing [18665.772600] Lustre: 466108:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.120@tcp: can't add NID 192.168.203.120@tcp130: rc = -75 [18665.786419] Lustre: 466108:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [18665.962306] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [18666.026510] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [18666.036930] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [18671.190714] Lustre: 466314:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.120@tcp: can't add NID 192.168.203.120@tcp130: rc = -75 [18671.202547] Lustre: 466314:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [18672.183092] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [18675.364342] LNetError: 465320:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.120 [18675.380873] LNetError: Protocol error connecting to 192.168.203.120@tcp117 at host 192.168.203.120:988: is it running a compatible version of Lustre? [18684.619879] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [18691.746755] LNetError: 462746:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.120 [18691.765274] LNetError: 462746:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 10 previous similar messages [18691.782770] LNetError: Protocol error connecting to 192.168.203.120@tcp115 at host 192.168.203.120:988: is it running a compatible version of Lustre? [18691.800665] LNetError: Skipped 10 previous similar messages [18692.927875] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [18702.653258] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [18703.104898] Lustre: lustre-OST0000: new disk, initializing [18703.114156] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [18703.120577] Lustre: 467283:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [18703.254806] Lustre: 467288:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.120@tcp: can't add NID 192.168.203.120@tcp130: rc = -75 [18703.262149] Lustre: 467288:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [18703.317827] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [18705.126956] LNet: 1 local NIs in recovery (showing 1): 192.168.203.120@tcp123 [18709.090822] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [18709.105760] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [18709.307484] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [18712.675842] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [18724.388481] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [18724.525737] LNetError: 462747:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.120 [18724.555590] LNetError: 462747:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 16 previous similar messages [18724.578452] LNetError: Protocol error connecting to 192.168.203.120@tcp127 at host 192.168.203.120:988: is it running a compatible version of Lustre? [18724.590036] LNetError: Skipped 16 previous similar messages [18735.080535] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [18735.095977] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [18735.107904] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [18735.124217] Lustre: Skipped 1 previous similar message [18740.236320] Lustre: server umount lustre-OST0000 complete [18747.872338] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [18747.884784] LustreError: Skipped 1 previous similar message [18747.892794] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [18747.918598] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [18747.920873] Lustre: Skipped 1 previous similar message [18750.433396] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [18750.435049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [18750.483394] Lustre: Skipped 1 previous similar message [18752.723654] Lustre: server umount lustre-MDT0000 complete [18757.978848] LustreError: 465298:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785952458 with bad export cookie 10673157611707702846 [18757.989868] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [18758.615431] Lustre: server umount lustre-MDT0001 complete [18779.047276] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [18783.885639] Key type lgssc unregistered [18784.618489] LNet: 469214:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [18784.633915] LNetError: 469214:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [18784.651727] LNet: Removed LNI 192.168.203.120@tcp [18785.695131] LNet: Removed LNI 192.168.203.120@tcp100 [18785.700818] LNetError: 469214:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [18785.710498] LNetError: 469214:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [18785.724859] LNet: Removed LNI 192.168.203.120@tcp101 [18786.784758] LNet: Removed LNI 192.168.203.120@tcp102 [18786.803452] LNetError: 469214:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [18786.813447] LNetError: 469214:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [18786.831247] LNet: Removed LNI 192.168.203.120@tcp103 [18787.871650] LNet: Removed LNI 192.168.203.120@tcp104 [18788.895264] LNet: Removed LNI 192.168.203.120@tcp105 [18788.897532] LNetError: 469214:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [18788.905305] LNetError: 469214:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [18788.913447] LNet: Removed LNI 192.168.203.120@tcp106 [18789.983240] LNet: Removed LNI 192.168.203.120@tcp107 [18791.007137] LNet: Removed LNI 192.168.203.120@tcp108 [18792.031171] LNet: Removed LNI 192.168.203.120@tcp109 [18793.119448] LNet: Removed LNI 192.168.203.120@tcp110 [18793.125860] LNetError: 469214:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [18793.137938] LNetError: 469214:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [18793.144566] LNet: Removed LNI 192.168.203.120@tcp111 [18794.207215] LNet: Removed LNI 192.168.203.120@tcp112 [18795.231165] LNet: Removed LNI 192.168.203.120@tcp113 [18796.255268] LNet: Removed LNI 192.168.203.120@tcp114 [18797.279142] LNet: Removed LNI 192.168.203.120@tcp115 [18798.303140] LNet: Removed LNI 192.168.203.120@tcp116 [18799.327156] LNet: Removed LNI 192.168.203.120@tcp117 [18800.352929] LNet: Removed LNI 192.168.203.120@tcp118 [18801.375143] LNet: Removed LNI 192.168.203.120@tcp119 [18801.377059] LNetError: 469214:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [18801.398042] LNetError: 469214:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [18801.423975] LNet: Removed LNI 192.168.203.120@tcp120 [18802.463134] LNet: Removed LNI 192.168.203.120@tcp121 [18803.487143] LNet: Removed LNI 192.168.203.120@tcp122 [18803.489972] LNet: Removed LNI 192.168.203.120@tcp123 [18804.511148] LNet: Removed LNI 192.168.203.120@tcp124 [18805.535144] LNet: Removed LNI 192.168.203.120@tcp125 [18806.559253] LNet: Removed LNI 192.168.203.120@tcp126 [18807.586135] LNet: Removed LNI 192.168.203.120@tcp127 [18808.607117] LNet: Removed LNI 192.168.203.120@tcp128 [18809.631134] LNet: Removed LNI 192.168.203.120@tcp129 [18809.633324] LNet: Removed LNI 192.168.203.120@tcp130 [18809.637807] LNet: Removed LNI 192.168.203.120@tcp131 [18809.652613] LNet: Removed LNI 192.168.203.120@tcp132 [18809.660431] LNet: Removed LNI 192.168.203.120@tcp133 [18809.668276] LNet: Removed LNI 192.168.203.120@tcp134 [18809.681668] LNet: Removed LNI 192.168.203.120@tcp135 [18810.968808] Key type .llcrypt unregistered [18810.976317] Key type ._llcrypt unregistered [18840.221279] Key type ._llcrypt registered [18840.226139] Key type .llcrypt registered [18840.418690] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [18860.591935] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [18862.066199] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [18862.284993] alg: No test for adler32 (adler32-zlib) [18863.463023] Lustre: Lustre: Build Version: 2.17.56_50_g82e980f [18864.074965] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [18866.068846] Key type lgssc registered [18868.262696] Lustre: Echo OBD driver; http://www.lustre.org/ [18926.572546] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 13:57:05 (1785952625) [18949.598264] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [18953.738275] Key type lgssc unregistered [18954.246031] LNet: 473311:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [18954.252270] LNetError: 473311:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [18954.271564] LNet: Removed LNI 192.168.203.120@tcp [18955.081490] Key type .llcrypt unregistered [18955.083614] Key type ._llcrypt unregistered [18977.726933] Key type ._llcrypt registered [18977.730037] Key type .llcrypt registered [18977.940894] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [18979.980815] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [18980.019051] alg: No test for adler32 (adler32-zlib) [18981.453527] Lustre: Lustre: Build Version: 2.17.56_50_g82e980f [18981.881639] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [18983.689367] Key type lgssc registered [18986.293831] Lustre: Echo OBD driver; http://www.lustre.org/ [19004.994367] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [19005.040632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [19006.627640] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [19006.733777] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [19006.881489] Lustre: lustre-MDT0000: new disk, initializing [19007.004421] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [19007.029189] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [19013.050657] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [19026.941257] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [19027.110284] Lustre: 474533:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [19027.208874] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [19027.216913] Lustre: Skipped 1 previous similar message [19027.416926] Lustre: lustre-MDT0001: new disk, initializing [19027.636933] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [19027.703187] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [19027.724559] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [19033.818559] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [19045.350244] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [19053.038524] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [19061.109910] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [19061.596111] Lustre: lustre-OST0000: new disk, initializing [19061.603313] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [19061.654869] Lustre: 476504:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [19061.821337] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [19067.958828] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [19067.978910] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [19068.140123] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [19069.808931] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [19082.998681] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [19089.171559] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing lnet_if_list [19099.950820] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [19100.018690] LNet: Added LNI 192.168.203.120@tcp10 [8/256/0/180] [19109.155497] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [19109.241237] LNet: Added LNI 192.168.203.120@tcp11 [8/256/0/180] [19118.163707] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [19118.204037] LNet: Added LNI 192.168.203.120@tcp12 [8/256/0/180] [19125.098493] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [19125.146860] LNet: Added LNI 192.168.203.120@tcp13 [8/256/0/180] [19131.435721] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [19131.477261] LNet: Added LNI 192.168.203.120@tcp14 [8/256/0/180] [19137.393195] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [19137.417678] LNet: Added LNI 192.168.203.120@tcp15 [8/256/0/180] [19142.921718] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [19142.961565] LNet: Added LNI 192.168.203.120@tcp16 [8/256/0/180] [19149.317969] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [19149.358079] LNet: Added LNI 192.168.203.120@tcp17 [8/256/0/180] [19156.966452] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [19157.017232] LNet: Added LNI 192.168.203.120@tcp18 [8/256/0/180] [19165.773315] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [19165.817784] LNet: Added LNI 192.168.203.120@tcp19 [8/256/0/180] [19174.532530] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [19174.563447] LNet: Added LNI 192.168.203.120@tcp20 [8/256/0/180] [19180.959440] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [19181.040276] LNet: Added LNI 192.168.203.120@tcp21 [8/256/0/180] [19188.541847] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [19188.613829] LNet: Added LNI 192.168.203.120@tcp22 [8/256/0/180] [19197.715176] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [19197.790356] LNet: Added LNI 192.168.203.120@tcp23 [8/256/0/180] [19209.520710] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [19209.538243] LNet: Added LNI 192.168.203.120@tcp24 [8/256/0/180] [19221.823172] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [19221.928226] LNet: Added LNI 192.168.203.120@tcp25 [8/256/0/180] [19234.628669] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [19234.748950] LNet: Added LNI 192.168.203.120@tcp26 [8/256/0/180] [19246.508178] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [19246.552987] LNet: Added LNI 192.168.203.120@tcp27 [8/256/0/180] [19255.720301] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [19255.758492] LNet: Added LNI 192.168.203.120@tcp28 [8/256/0/180] [19266.011781] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [19266.075840] LNet: Added LNI 192.168.203.120@tcp29 [8/256/0/180] [19276.568291] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [19276.604137] LNet: Added LNI 192.168.203.120@tcp30 [8/256/0/180] [19286.458500] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [19286.506434] LNet: Added LNI 192.168.203.120@tcp31 [8/256/0/180] [19295.031477] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [19295.088468] LNet: Added LNI 192.168.203.120@tcp32 [8/256/0/180] [19305.004707] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [19305.047124] LNet: Added LNI 192.168.203.120@tcp33 [8/256/0/180] [19314.941937] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [19314.968742] LNet: Added LNI 192.168.203.120@tcp34 [8/256/0/180] [19324.597343] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [19324.632620] LNet: Added LNI 192.168.203.120@tcp35 [8/256/0/180] [19334.182849] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [19334.254047] LNet: Added LNI 192.168.203.120@tcp36 [8/256/0/180] [19348.061205] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [19348.156110] LNet: Added LNI 192.168.203.120@tcp37 [8/256/0/180] [19362.238517] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [19362.297995] LNet: Added LNI 192.168.203.120@tcp38 [8/256/0/180] [19375.251801] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [19375.305739] LNet: Added LNI 192.168.203.120@tcp39 [8/256/0/180] [19388.202857] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [19388.266761] LNet: Added LNI 192.168.203.120@tcp40 [8/256/0/180] [19397.499420] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [19397.553213] LNet: Added LNI 192.168.203.120@tcp41 [8/256/0/180] [19411.931873] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [19412.034359] LNet: Added LNI 192.168.203.120@tcp42 [8/256/0/180] [19424.759772] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [19424.825986] LNet: Added LNI 192.168.203.120@tcp43 [8/256/0/180] [19436.604693] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [19436.630796] LNet: Added LNI 192.168.203.120@tcp44 [8/256/0/180] [19447.388387] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [19447.430365] LNet: Added LNI 192.168.203.120@tcp45 [8/256/0/180] [19459.232455] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [19459.308582] LNet: Added LNI 192.168.203.120@tcp46 [8/256/0/180] [19472.204559] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [19472.274574] LNet: Added LNI 192.168.203.120@tcp47 [8/256/0/180] [19484.338810] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [19484.412440] LNet: Added LNI 192.168.203.120@tcp48 [8/256/0/180] [19494.240893] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [19494.280573] LNet: Added LNI 192.168.203.120@tcp49 [8/256/0/180] [19505.797483] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [19505.830639] LNet: Added LNI 192.168.203.120@tcp50 [8/256/0/180] [19534.335871] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19534.341714] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [19534.376523] Lustre: Skipped 2 previous similar messages [19534.391170] Lustre: Skipped 2 previous similar messages [19539.225873] Lustre: server umount lustre-MDT0000 complete [19543.800771] LustreError: 476503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785953244 with bad export cookie 9471586112401229314 [19543.804797] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [19543.816696] LustreError: 476503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [19544.284450] Lustre: server umount lustre-MDT0001 complete [19560.927348] Lustre: 473802:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785953245/real 1785953245] req@00000000e668c8e7 x1872707119708032/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785953261 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [19560.978091] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19565.023945] Lustre: 473801:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785953249/real 1785953249] req@00000000f0a767d9 x1872707119708160/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785953265 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [19566.049604] Lustre: 473800:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785953250/real 1785953250] req@00000000e7307198 x1872707119708288/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785953266 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [19570.154178] Lustre: 473802:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785953254/real 1785953254] req@0000000089827dcc x1872707119708544/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785953270 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [19572.477360] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [19575.327096] Lustre: 473802:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785953259/real 1785953259] req@00000000165a996f x1872707119708672/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785953275 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [19587.561134] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [19588.959528] LustreError: 476494:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [19588.988849] LustreError: 476494:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [19589.046633] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [19590.053091] LustreError: 476495:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [19590.080474] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [19590.103172] LustreError: 476495:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [19595.231970] LustreError: 476494:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [19598.042992] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [19600.376236] LustreError: 476495:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [19605.481640] LustreError: 476494:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [19605.502028] LustreError: 476494:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [19609.174372] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [19609.761064] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [19615.220710] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [19615.506560] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [19624.646717] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [19632.021104] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [19660.771908] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [19660.783471] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19660.789779] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [19660.808266] Lustre: Skipped 1 previous similar message [19663.801665] Lustre: server umount lustre-OST0000 complete [19671.008028] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [19671.013935] LustreError: Skipped 1 previous similar message [19671.020053] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19671.038092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [19671.047389] Lustre: Skipped 3 previous similar messages [19676.357925] Lustre: server umount lustre-MDT0000 complete [19676.645171] LustreError: 501754:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [19676.674737] LustreError: 501754:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [19680.800187] LustreError: 474524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785953381 with bad export cookie 9471586112401232170 [19680.819396] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [19680.832647] LustreError: 474524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [19681.811421] Lustre: server umount lustre-MDT0001 complete [19697.796154] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [19701.537989] Key type lgssc unregistered [19702.064849] LNet: 503615:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [19702.082876] LNetError: 503615:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [19702.096724] LNet: Removed LNI 192.168.203.120@tcp [19702.099652] LNet: Removed LNI 192.168.203.120@tcp10 [19702.106634] LNet: Removed LNI 192.168.203.120@tcp11 [19702.114480] LNet: Removed LNI 192.168.203.120@tcp12 [19702.118656] LNet: Removed LNI 192.168.203.120@tcp13 [19702.138179] LNet: Removed LNI 192.168.203.120@tcp14 [19702.144274] LNet: Removed LNI 192.168.203.120@tcp15 [19702.152728] LNet: Removed LNI 192.168.203.120@tcp16 [19702.161703] LNet: Removed LNI 192.168.203.120@tcp17 [19702.163356] LNet: Removed LNI 192.168.203.120@tcp18 [19702.164862] LNet: Removed LNI 192.168.203.120@tcp19 [19702.172521] LNet: Removed LNI 192.168.203.120@tcp20 [19702.177603] LNet: Removed LNI 192.168.203.120@tcp21 [19702.185071] LNet: Removed LNI 192.168.203.120@tcp22 [19702.190040] LNet: Removed LNI 192.168.203.120@tcp23 [19702.191709] LNet: Removed LNI 192.168.203.120@tcp24 [19702.195572] LNet: Removed LNI 192.168.203.120@tcp25 [19702.202140] LNet: Removed LNI 192.168.203.120@tcp26 [19702.210784] LNet: Removed LNI 192.168.203.120@tcp27 [19702.219061] LNet: Removed LNI 192.168.203.120@tcp28 [19702.223935] LNet: Removed LNI 192.168.203.120@tcp29 [19702.228952] LNet: Removed LNI 192.168.203.120@tcp30 [19702.237294] LNet: Removed LNI 192.168.203.120@tcp31 [19702.239749] LNet: Removed LNI 192.168.203.120@tcp32 [19702.247597] LNet: Removed LNI 192.168.203.120@tcp33 [19702.253206] LNet: Removed LNI 192.168.203.120@tcp34 [19702.256620] LNet: Removed LNI 192.168.203.120@tcp35 [19702.261542] LNet: Removed LNI 192.168.203.120@tcp36 [19702.263099] LNet: Removed LNI 192.168.203.120@tcp37 [19702.270919] LNet: Removed LNI 192.168.203.120@tcp38 [19702.272779] LNet: Removed LNI 192.168.203.120@tcp39 [19702.281697] LNet: Removed LNI 192.168.203.120@tcp40 [19702.287655] LNet: Removed LNI 192.168.203.120@tcp41 [19702.293163] LNet: Removed LNI 192.168.203.120@tcp42 [19702.299667] LNet: Removed LNI 192.168.203.120@tcp43 [19702.306902] LNet: Removed LNI 192.168.203.120@tcp44 [19702.313273] LNet: Removed LNI 192.168.203.120@tcp45 [19702.321742] LNet: Removed LNI 192.168.203.120@tcp46 [19702.327544] LNet: Removed LNI 192.168.203.120@tcp47 [19702.335079] LNet: Removed LNI 192.168.203.120@tcp48 [19702.341855] LNet: Removed LNI 192.168.203.120@tcp49 [19702.365224] LNet: Removed LNI 192.168.203.120@tcp50 [19703.855338] Key type .llcrypt unregistered [19703.859743] Key type ._llcrypt unregistered [19733.874394] Key type ._llcrypt registered [19733.878964] Key type .llcrypt registered [19734.078032] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [19756.833758] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [19758.411246] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [19758.713373] alg: No test for adler32 (adler32-zlib) [19760.104308] Lustre: Lustre: Build Version: 2.17.56_50_g82e980f [19760.600124] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [19762.695236] Key type lgssc registered [19765.385208] Lustre: Echo OBD driver; http://www.lustre.org/ [19817.533964] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [19817.558829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [19819.135579] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [19819.196867] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [19819.369264] Lustre: lustre-MDT0000: new disk, initializing [19819.543143] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [19819.594538] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [19827.585338] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [19841.698971] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [19841.884429] Lustre: 506815:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [19842.055574] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [19842.067382] Lustre: Skipped 1 previous similar message [19842.282741] Lustre: lustre-MDT0001: new disk, initializing [19842.511532] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [19842.637139] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [19842.671746] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [19848.473654] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [19861.199578] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [19870.581325] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [19884.150070] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [19884.804970] Lustre: *** cfs_fail_loc=90f, val=0*** [19885.052042] Lustre: lustre-OST0000: new disk, initializing [19885.058460] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [19885.079957] Lustre: 508834:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [19885.278591] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [19892.932068] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [19892.940681] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [19893.128169] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [19896.681968] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [19909.755476] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [19923.939309] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19923.960216] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [19923.963546] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [19923.974666] Lustre: Skipped 1 previous similar message [19924.010524] LustreError: Skipped 1 previous similar message [19926.798415] Lustre: server umount lustre-OST0000 complete [19934.178965] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19934.180258] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [19934.202880] Lustre: Skipped 1 previous similar message [19934.256531] Lustre: Skipped 2 previous similar messages [19939.301632] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [19939.317656] Lustre: Skipped 1 previous similar message [19940.468319] Lustre: server umount lustre-MDT0000 complete [19947.346946] LustreError: 507620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785953647 with bad export cookie 7037978801160504466 [19947.362793] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [19947.379378] LustreError: 507620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [19947.960358] Lustre: server umount lustre-MDT0001 complete [19964.558818] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [19968.247400] Key type lgssc unregistered [19968.591507] LNet: 510749:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [19968.604100] LNetError: 510749:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [19968.618680] LNet: Removed LNI 192.168.203.120@tcp [19969.862745] Key type .llcrypt unregistered [19969.865325] Key type ._llcrypt unregistered [19997.089321] Key type ._llcrypt registered [19997.090758] Key type .llcrypt registered [19997.345897] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [19999.337018] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [19999.462274] alg: No test for adler32 (adler32-zlib) [20000.864479] Lustre: Lustre: Build Version: 2.17.56_50_g82e980f [20001.264213] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [20002.975176] Key type lgssc registered [20004.488086] Lustre: Echo OBD driver; http://www.lustre.org/ [20016.811984] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [20016.901787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [20019.235752] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [20027.456393] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [20029.427358] LustreError: 512022:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [20029.456445] LustreError: 512022:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [20034.539029] LustreError: 512021:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [20039.145336] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [20039.665283] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [20039.927017] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [20045.730442] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [20056.370607] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [20064.268909] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [20074.324338] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [20074.986681] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [20085.932620] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [20096.811530] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [20116.623114] loop0: detected capacity change from 0 to 204800000 [20116.800065] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [20116.809386] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [20116.831807] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [20128.828973] loop0: detected capacity change from 0 to 204800000 [20129.185434] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [20129.225230] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [20129.266429] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [20141.784916] loop0: detected capacity change from 0 to 204800000 [20149.978091] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [20150.121909] Lustre: fs15246-OST0000: new disk, initializing [20150.126088] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [20150.139166] Lustre: 515626:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [20150.282116] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [20159.003463] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [20173.786850] loop1: detected capacity change from 0 to 204800000 [20182.371495] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [20182.575640] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [20182.634978] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [20182.779471] Lustre: fs15246-MDT0000: new disk, initializing [20183.022360] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [20183.086573] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [20183.305702] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [20184.355901] Lustre: 516750:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@000000000a23bdd2 x1872708188038400/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1785953900 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [20189.705745] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [20193.451937] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [20193.471503] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [20193.521677] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [20220.389950] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [20220.430203] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [20225.517616] LustreError: 513866:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [20225.833859] Lustre: server umount fs15246-MDT0000 complete [20230.645604] LustreError: 512017:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [20235.750933] LustreError: 512016:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [20238.695543] Lustre: server umount fs15246-OST0000 complete [20256.239262] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [20256.243830] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [20256.250409] Lustre: Skipped 1 previous similar message [20256.278165] Lustre: Skipped 1 previous similar message [20261.344043] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [20266.467032] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [20266.474393] Lustre: Skipped 2 previous similar messages [20270.048148] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [20270.319808] Lustre: server umount lustre-OST0000 complete [20276.705616] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [20276.712791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [20276.731874] Lustre: Skipped 1 previous similar message [20276.759938] Lustre: Skipped 1 previous similar message [20282.223972] Lustre: server umount lustre-MDT0000 complete [20286.948106] LustreError: 513136:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [20286.965281] LustreError: 513136:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [20287.022314] LustreError: 512757:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785953987 with bad export cookie 942380497416554617 [20287.028572] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [20287.036199] LustreError: 512757:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [20287.487739] Lustre: server umount lustre-MDT0001 complete [20304.847079] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [20311.110957] Key type lgssc unregistered [20311.599407] LNet: 519857:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [20311.627564] LNetError: 519857:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [20311.714566] LNet: Removed LNI 192.168.203.120@tcp [20314.409060] Key type .llcrypt unregistered [20314.418354] Key type ._llcrypt unregistered [20348.608886] Key type ._llcrypt registered [20348.614438] Key type .llcrypt registered [20348.812766] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [20350.641329] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [20350.792211] alg: No test for adler32 (adler32-zlib) [20352.094761] Lustre: Lustre: Build Version: 2.17.56_50_g82e980f [20352.530670] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [20354.408094] Key type lgssc registered [20356.262567] Lustre: Echo OBD driver; http://www.lustre.org/ [20409.810811] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [20426.943336] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [20487.138443] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [20487.188870] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [20488.692113] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [20488.761840] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [20488.982609] Lustre: lustre-MDT0000: new disk, initializing [20489.168396] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [20489.205977] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [20498.135177] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [20515.571837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [20515.803858] Lustre: 524357:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [20515.976428] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [20515.988173] Lustre: Skipped 1 previous similar message [20516.193437] Lustre: lustre-MDT0001: new disk, initializing [20516.386874] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [20516.525320] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [20516.548588] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [20523.624251] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [20537.223479] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [20546.884455] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [20558.681650] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [20559.223081] Lustre: lustre-OST0000: new disk, initializing [20559.228367] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [20559.245446] Lustre: 526330:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [20559.377563] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [20565.605836] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [20565.627522] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [20565.744483] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [20569.529465] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [20581.385060] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [20587.546125] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [20587.740923] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [20594.407931] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [20594.830563] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [20601.836507] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [20601.850722] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [20601.860081] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [20601.880048] LustreError: Skipped 1 previous similar message [20601.913260] Lustre: Skipped 1 previous similar message [20603.729604] Lustre: server umount lustre-OST0000 complete [20612.073864] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [20612.096391] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [20612.104367] Lustre: Skipped 1 previous similar message [20617.186442] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [20617.205165] Lustre: Skipped 2 previous similar messages [20622.306970] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [20622.309546] Lustre: Skipped 1 previous similar message [20623.327091] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [20623.923367] Lustre: server umount lustre-MDT0000 complete [20628.343831] LustreError: 527874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785954328 with bad export cookie 7863259437837709997 [20628.358629] LustreError: 527874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [20628.365571] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [20628.921705] Lustre: server umount lustre-MDT0001 complete [20661.191797] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [20674.824956] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [20718.013843] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 20290 sec ========= 14:26:55 (1785954415) [20720.848341] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 14:26:58 (1785954418) === [20726.433077] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 14:27:04 (1785954424) === [20781.091750] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [20785.523793] Key type lgssc unregistered [20786.122713] LNet: 532989:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [20786.131281] LNetError: 532989:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [20786.173472] LNet: Removed LNI 192.168.203.120@tcp [20788.154849] Key type .llcrypt unregistered [20788.156695] Key type ._llcrypt unregistered