[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 478048293 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002372] x2apic enabled [ 0.003006] Switched APIC routing to physical x2apic. [ 0.004011] kvm-guest: setup PV IPIs [ 0.007512] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009014] pid_max: default: 32768 minimum: 301 [ 0.010140] LSM: Security Framework initializing [ 0.011049] Yama: becoming mindful. [ 0.012034] SELinux: Initializing. [ 0.013056] *** VALIDATE selinux *** [ 0.021594] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026565] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027149] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028110] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029109] *** VALIDATE tmpfs *** [ 0.030444] *** VALIDATE proc *** [ 0.031244] *** VALIDATE cgroup *** [ 0.032008] *** VALIDATE cgroup2 *** [ 0.033236] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034141] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036029] Spectre V2 : User space: Vulnerable [ 0.037010] Speculative Store Bypass: Vulnerable [ 0.040400] debug: unmapping init [mem 0xffffffffbb259000-0xffffffffbb260fff] [ 0.042852] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043650] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044031] ... version: 2 [ 0.045012] ... bit width: 48 [ 0.046010] ... generic registers: 4 [ 0.047010] ... value mask: 0000ffffffffffff [ 0.048012] ... max period: 00007fffffffffff [ 0.049014] ... fixed-purpose events: 3 [ 0.050010] ... event mask: 000000070000000f [ 0.051313] rcu: Hierarchical SRCU implementation. [ 0.053328] smp: Bringing up secondary CPUs ... [ 0.054517] x86: Booting SMP configuration: [ 0.055025] .... node #0, CPUs: #1 #2 #3 [ 0.058143] smp: Brought up 1 node, 4 CPUs [ 0.060011] smpboot: Max logical packages: 1 [ 0.061016] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.131921] node 0 deferred pages initialised in 68ms [ 0.133194] devtmpfs: initialized [ 0.134222] x86/mm: Memory block size: 128MB [ 0.136737] gcov: version magic: 0x41383552 [ 0.138326] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.139138] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.140291] pinctrl core: initialized pinctrl subsystem [ 0.141183] [ 0.141700] ************************************************************* [ 0.142014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.143016] ** ** [ 0.144019] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.145013] ** ** [ 0.146014] ** This means that this kernel is built to expose internal ** [ 0.147012] ** IOMMU data structures, which may compromise security on ** [ 0.148014] ** your system. ** [ 0.149017] ** ** [ 0.150020] ** If you see this message and you are not debugging the ** [ 0.151013] ** kernel, report this immediately to your vendor! ** [ 0.152011] ** ** [ 0.153012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.154016] ************************************************************* [ 0.155766] NET: Registered protocol family 16 [ 0.156455] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.157079] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.158088] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.160113] cpuidle: using governor menu [ 0.161683] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.162421] PCI: Using configuration type 1 for base access [ 0.163138] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.171072] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.172051] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.174100] cryptd: max_cpu_qlen set to 1000 [ 0.178256] ACPI: Added _OSI(Module Device) [ 0.180013] ACPI: Added _OSI(Processor Device) [ 0.182014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.183012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.189573] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.196219] ACPI: Interpreter enabled [ 0.197063] ACPI: PM: (supports S0 S3 S4 S5) [ 0.198011] ACPI: Using IOAPIC for interrupt routing [ 0.199100] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.200372] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.209661] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.210048] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.211023] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.212078] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.214323] acpiphp: Slot [2] registered [ 0.215103] acpiphp: Slot [5] registered [ 0.216101] acpiphp: Slot [6] registered [ 0.217118] acpiphp: Slot [7] registered [ 0.218133] acpiphp: Slot [8] registered [ 0.219110] acpiphp: Slot [9] registered [ 0.220100] acpiphp: Slot [10] registered [ 0.221000] acpiphp: Slot [3] registered [ 0.222065] acpiphp: Slot [4] registered [ 0.224442] acpiphp: Slot [11] registered [ 0.226139] acpiphp: Slot [12] registered [ 0.228098] acpiphp: Slot [13] registered [ 0.230111] acpiphp: Slot [14] registered [ 0.232109] acpiphp: Slot [15] registered [ 0.233234] acpiphp: Slot [16] registered [ 0.235133] acpiphp: Slot [17] registered [ 0.237109] acpiphp: Slot [18] registered [ 0.239094] acpiphp: Slot [19] registered [ 0.240073] acpiphp: Slot [20] registered [ 0.241133] acpiphp: Slot [21] registered [ 0.242095] acpiphp: Slot [22] registered [ 0.244187] acpiphp: Slot [23] registered [ 0.246124] acpiphp: Slot [24] registered [ 0.247092] acpiphp: Slot [25] registered [ 0.249099] acpiphp: Slot [26] registered [ 0.251169] acpiphp: Slot [27] registered [ 0.252096] acpiphp: Slot [28] registered [ 0.254103] acpiphp: Slot [29] registered [ 0.256115] acpiphp: Slot [30] registered [ 0.257110] acpiphp: Slot [31] registered [ 0.259080] PCI host bridge to bus 0000:00 [ 0.261074] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.263037] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.266024] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.269024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.272026] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.275025] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.277279] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.280200] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.284435] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.295764] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.301063] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.304032] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.306018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.310030] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.313106] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.316022] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.319059] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.322961] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.328022] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.339900] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.345015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.350448] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.359017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.367020] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.386016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.401915] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.421016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.435019] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.455019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.468335] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.475016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.483018] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.502017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.513232] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.523021] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.534019] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.555020] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.565733] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.572018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.577019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.591015] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.599000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.609025] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.616020] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.642018] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.656859] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.659290] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.660248] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.662298] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.664624] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.668337] iommu: Default domain type: Passthrough [ 0.670632] SCSI subsystem initialized [ 0.672136] ACPI: bus type USB registered [ 0.674141] usbcore: registered new interface driver usbfs [ 0.676107] usbcore: registered new interface driver hub [ 0.677090] usbcore: registered new device driver usb [ 0.679183] pps_core: LinuxPPS API ver. 1 registered [ 0.681010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.684071] PTP clock support registered [ 0.686068] EDAC MC: Ver: 3.0.0 [ 0.687111] PCI: Using ACPI for IRQ routing [ 0.688000] NetLabel: Initializing [ 0.688000] NetLabel: domain hash size = 128 [ 0.690012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.691068] NetLabel: unlabeled traffic allowed by default [ 0.693035] vgaarb: loaded [ 0.693699] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.695013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.701000] clocksource: Switched to clocksource kvm-clock [ 0.807452] VFS: Disk quotas dquot_6.6.0 [ 0.809013] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.811585] *** VALIDATE ramfs *** [ 0.812795] *** VALIDATE hugetlbfs *** [ 0.814180] pnp: PnP ACPI init [ 0.816327] pnp: PnP ACPI: found 6 devices [ 0.859764] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.862787] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.864901] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.867065] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.869384] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.871599] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.874302] NET: Registered protocol family 2 [ 0.876652] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.881326] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.884774] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.890786] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.894520] TCP: Hash tables configured (established 65536 bind 65536) [ 0.897745] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.900891] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.903938] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.907348] NET: Registered protocol family 1 [ 0.911499] RPC: Registered named UNIX socket transport module. [ 0.913776] RPC: Registered udp transport module. [ 0.915551] RPC: Registered tcp transport module. [ 0.917098] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.918728] NET: Registered protocol family 44 [ 0.919855] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.921801] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.924130] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.926278] PCI: CLS 0 bytes, default 64 [ 0.928126] Unpacking initramfs... [ 2.284533] debug: unmapping init [mem 0xffff9b7a7cc54000-0xffff9b7a7ffbffff] [ 2.287291] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.288801] software IO TLB: mapped [mem 0x00000000a3600000-0x00000000a7600000] (64MB) [ 2.291244] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.788720] Initialise system trusted keyrings [ 2.790510] Key type blacklist registered [ 2.792650] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.801667] zbud: loaded [ 2.804752] *** VALIDATE nfs *** [ 2.805741] *** VALIDATE nfs4 *** [ 2.807221] pstore: using deflate compression [ 2.810324] Platform Keyring initialized [ 2.928986] NET: Registered protocol family 38 [ 2.930850] Key type asymmetric registered [ 2.932499] Asymmetric key parser 'x509' registered [ 2.934313] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.937410] io scheduler mq-deadline registered [ 2.939207] io scheduler kyber registered [ 2.940867] io scheduler bfq registered [ 2.943184] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.946217] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.949395] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.952501] ACPI: Power Button [PWRF] [ 2.957565] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.963855] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.978470] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.986317] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.000342] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.026941] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.055073] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.059688] Non-volatile memory driver v1.3 [ 3.061403] Linux agpgart interface v0.103 [ 3.094785] virtio_blk virtio1: [vda] 136528 512-byte logical blocks (69.9 MB/66.7 MiB) [ 3.097986] vda: detected capacity change from 0 to 69902336 [ 3.120469] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.123214] vdb: detected capacity change from 0 to 1073741824 [ 3.141855] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.145076] vdc: detected capacity change from 0 to 2621440000 [ 3.165765] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.168785] vdd: detected capacity change from 0 to 2621440000 [ 3.195900] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.198742] vde: detected capacity change from 0 to 4294967296 [ 3.230634] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.233913] vdf: detected capacity change from 0 to 4294967296 [ 3.261222] libphy: Fixed MDIO Bus: probed [ 3.278982] usbcore: registered new interface driver usbserial_generic [ 3.281636] usbserial: USB Serial support registered for generic [ 3.284228] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.287803] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.289689] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.292723] mousedev: PS/2 mouse device common for all mice [ 3.295334] rtc_cmos 00:05: RTC can wake from S4 [ 3.296747] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.299833] rtc_cmos 00:05: registered as rtc0 [ 3.302943] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.303552] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.306287] intel_pstate: CPU model not supported [ 3.309890] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.313673] hid: raw HID events driver (C) Jiri Kosina [ 3.315800] usbcore: registered new interface driver usbhid [ 3.317853] usbhid: USB HID core driver [ 3.319347] drop_monitor: Initializing network drop monitor service [ 3.321775] Initializing XFRM netlink socket [ 3.323739] NET: Registered protocol family 10 [ 3.326472] Segment Routing with IPv6 [ 3.327941] NET: Registered protocol family 17 [ 3.330000] mpls_gso: MPLS GSO support [ 3.337227] RAS: Correctable Errors collector initialized. [ 3.339325] AVX version of gcm_enc/dec engaged. [ 3.340864] AES CTR mode by8 optimization enabled [ 3.432579] sched_clock: Marking stable (3432550903, 0)->(4372929138, -940378235) [ 3.436216] registered taskstats version 1 [ 3.439111] Loading compiled-in X.509 certificates [ 3.440700] zswap: loaded using pool lzo/zbud [ 3.464057] Key type big_key registered [ 3.474874] Key type encrypted registered [ 3.475899] ima: No TPM chip found, activating TPM-bypass! [ 3.477165] ima: Allocated hash algorithm: sha1 [ 3.478635] ima: No architecture policies found [ 3.480201] evm: Initialising EVM extended attributes: [ 3.481722] evm: security.selinux [ 3.482805] evm: security.ima [ 3.483831] evm: security.capability [ 3.484922] evm: HMAC attrs: 0x1 [ 3.487328] rtc_cmos 00:05: setting system clock to 2026-06-12 01:34:37 UTC (1781228077) [ 3.492713] debug: unmapping init [mem 0xffffffffbc203000-0xffffffffbc3fffff] [ 3.495971] debug: unmapping init [mem 0xffffffffbaf82000-0xffffffffbb258fff] [ 3.508059] Write protecting the kernel read-only data: 28672k [ 3.510974] debug: unmapping init [mem 0xffffffffb9603000-0xffffffffb97fffff] [ 3.513196] debug: unmapping init [mem 0xffffffffb9f14000-0xffffffffb9ffffff] [ 3.543986] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.552289] systemd[1]: Detected virtualization kvm. [ 3.554219] systemd[1]: Detected architecture x86-64. [ 3.556181] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.578499] systemd[1]: No hostname configured. [ 3.580382] systemd[1]: Set hostname to . [ 3.582415] random: systemd: uninitialized urandom read (16 bytes read) [ 3.584492] systemd[1]: Initializing machine ID from random generator. [ 3.624846] random: ln: uninitialized urandom read (6 bytes read) [ 3.706968] random: systemd: uninitialized urandom read (16 bytes read) [ 3.709373] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.714773] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.719037] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... Starting Journal Service... Starting Create Volatile Files and Directories... [ OK ] Listening on udev Kernel Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Timers. Starting Setup Virtual Console... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.244597] device-mapper: uevent: version 1.0.3 [ 4.246768] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... [ 4.582425] random: fast init done Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.949909] virtio_net virtio0 ens2: renamed from eth0 [ 5.059279] scsi host0: ata_piix [ 5.062303] scsi host1: ata_piix [ 5.064470] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.067073] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.551943] dracut-initqueue[588]: RTNETLINK answers: File exists [ 9.990162] random: crng init done [ 9.991598] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.379604] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.467912] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.703320] SELinux: Disabled at runtime. [ 11.760439] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.768502] systemd[1]: Detected virtualization kvm. [ 11.771562] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.230928] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.234827] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.240317] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.244465] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.247876] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.256503] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.269186] systemd[1]: Mounting Huge Pages File System... Mounting Huge Pages File System... Mounting POSIX Message Queue File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-getty.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ 12.356797] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Kernel Debug File System... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Reached target Slices. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Remount Root and Kernel File Systems... Starting Apply Kernel Variables... [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.617825] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.845799] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.855318] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.971772] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.984946] EDAC sbridge: Ver: 1.1.2 [ 14.601603] Key type dns_resolver registered [ 14.903259] NFS: Registering the id_resolver key type [ 14.905385] Key type id_resolver registered [ 14.907105] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg254-server login: [ 42.626477] libcfs: loading out-of-tree module taints kernel. [ 42.645043] Key type ._llcrypt registered [ 42.646127] Key type .llcrypt registered [ 42.737330] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [ 51.958662] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 52.680781] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 52.687108] alg: No test for adler32 (adler32-zlib) [ 54.155077] Lustre: Lustre: Build Version: 2.17.53_64_gab2e0dd [ 55.289523] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 57.111374] Key type lgssc registered [ 59.200073] hrtimer: interrupt took 3974585 ns [ 59.505808] Lustre: Echo OBD driver; http://www.lustre.org/ [ 79.296304] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 124.680510] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 140.175602] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 140.219501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 141.587995] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 141.621951] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 141.766267] Lustre: lustre-MDT0000: new disk, initializing [ 141.875410] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 141.893511] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 146.903569] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 163.467451] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 163.709172] Lustre: 6480:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 163.762804] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 163.777433] Lustre: Skipped 1 previous similar message [ 163.989328] Lustre: lustre-MDT0001: new disk, initializing [ 164.091695] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 164.131065] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 164.154872] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 169.315737] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 174.873864] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 185.935156] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 186.124352] Lustre: lustre-OST0000: new disk, initializing [ 186.126737] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 186.131135] Lustre: 8383:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 186.203392] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 191.049724] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 191.065263] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 191.213387] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 193.198165] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 208.931024] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 209.080515] Lustre: lustre-OST0001: new disk, initializing [ 209.085869] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 209.091618] Lustre: 9439:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 209.162189] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 216.809803] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 219.217856] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 219.228895] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 219.278634] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 228.952414] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 235.489753] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 243.289421] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing check_logdir /tmp/testlogs/ [ 248.692306] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing yml_node [ 255.228701] Lustre: DEBUG MARKER: Client: 2.17.53.64 [ 257.627545] Lustre: DEBUG MARKER: MDS: 2.17.53.64 [ 261.795877] Lustre: DEBUG MARKER: OSS: 2.17.53.64 [ 264.133804] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Jun 11 21:38:55 EDT 2026 [ 284.564890] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 286.965781] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 306.144926] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 306.147939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 306.161326] Lustre: Skipped 3 previous similar messages [ 306.174095] Lustre: Skipped 3 previous similar messages [ 309.224761] Lustre: server umount lustre-MDT0000 complete [ 312.855366] LustreError: 6473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781228386 with bad export cookie 7637813191774018215 [ 312.876504] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 316.383927] LustreError: 6488:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 316.384076] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 316.414898] LustreError: 6488:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 316.420074] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 316.428459] Lustre: Skipped 1 previous similar message [ 316.456536] Lustre: Skipped 1 previous similar message [ 317.412207] LustreError: 6486:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 317.428698] LustreError: 6486:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 319.241579] Lustre: server umount lustre-MDT0001 complete [ 323.458585] Lustre: server umount lustre-OST0000 complete [ 327.446021] Lustre: server umount lustre-OST0001 complete [ 335.003090] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [ 342.860519] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 385.808642] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 386.064854] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 386.092856] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 386.164931] Lustre: lustre-MDT0000: new disk, initializing [ 386.265036] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 386.315960] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 391.472774] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 403.608192] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 403.789200] Lustre: 14955:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 403.836373] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 403.839691] Lustre: Skipped 1 previous similar message [ 403.902200] Lustre: lustre-MDT0001: new disk, initializing [ 404.002173] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 404.038402] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 404.053560] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 409.232635] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 418.058464] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 422.547464] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 431.172158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 431.542777] Lustre: lustre-OST0000: new disk, initializing [ 431.549936] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 431.565577] Lustre: 16697:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 431.677708] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 432.981104] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 432.998466] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 433.283531] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 438.548525] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 447.310700] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 454.700677] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 455.021559] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 460.644994] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 460.854287] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 463.843501] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 463.845444] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 463.862512] LustreError: Skipped 1 previous similar message [ 463.866660] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 463.866668] Lustre: Skipped 2 previous similar messages [ 463.911961] Lustre: Skipped 1 previous similar message [ 468.982205] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 468.991391] Lustre: Skipped 1 previous similar message [ 469.587915] Lustre: server umount lustre-OST0000 complete [ 474.082920] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 474.105849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 474.113023] Lustre: Skipped 2 previous similar messages [ 479.995575] Lustre: server umount lustre-MDT0000 complete [ 483.850847] LustreError: 15743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781228557 with bad export cookie 7637813191774019748 [ 483.854405] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 483.861478] LustreError: 15743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 484.180436] Lustre: server umount lustre-MDT0001 complete [ 489.627352] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 21:42:41 (1781228561) [ 492.867604] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 495.038688] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 21:42:47 (1781228567) [ 503.836882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 504.237011] LustreError: 18962:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 504.311051] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 509.206937] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 509.409878] LustreError: 18963:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 514.530970] LustreError: 18962:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 518.516843] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 523.970499] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 530.303739] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 536.030996] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 544.132951] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 551.823886] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 557.994307] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 570.346869] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 570.357640] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 570.378147] Lustre: Skipped 1 previous similar message [ 570.394415] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 570.406512] Lustre: Skipped 2 previous similar messages [ 573.427898] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 575.732452] Lustre: server umount lustre-OST0000 complete [ 580.578327] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 580.586311] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 580.598711] Lustre: Skipped 1 previous similar message [ 588.768826] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 588.786027] Lustre: Skipped 6 previous similar messages [ 594.399131] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 594.628975] Lustre: server umount lustre-MDT0000 complete [ 598.674926] LustreError: 19681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781228672 with bad export cookie 7637813191774020630 [ 598.681369] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 598.685924] LustreError: 19681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 598.969583] Lustre: server umount lustre-MDT0001 complete [ 610.638535] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 613.794743] Key type lgssc unregistered [ 614.107800] LNet: 22140:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 614.114964] LNetError: 22140:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 614.131991] LNet: Removed LNI 192.168.202.154@tcp [ 614.919254] Key type .llcrypt unregistered [ 614.921182] Key type ._llcrypt unregistered [ 641.714369] Key type ._llcrypt registered [ 641.716655] Key type .llcrypt registered [ 641.812794] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 642.985639] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 643.202268] alg: No test for adler32 (adler32-zlib) [ 644.362793] Lustre: Lustre: Build Version: 2.17.53_64_gab2e0dd [ 644.699766] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 646.431276] Key type lgssc registered [ 648.033214] Lustre: Echo OBD driver; http://www.lustre.org/ [ 658.907822] loop: module loaded [ 659.014174] loop0: detected capacity change from 0 to 204800000 [ 659.347933] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 659.356251] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 659.656082] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 667.479431] loop0: detected capacity change from 0 to 204800000 [ 667.655947] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 667.665573] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 667.688281] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 673.634445] loop0: detected capacity change from 0 to 204800000 [ 679.634372] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 679.662924] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 681.024110] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 681.081596] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 681.152398] Lustre: lustre-MDT0000: new disk, initializing [ 681.282616] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 681.300093] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 685.187191] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 694.441261] loop1: detected capacity change from 0 to 204800000 [ 700.147246] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 700.506897] Lustre: lustre-OST07c6: new disk, initializing [ 700.513511] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 700.527345] Lustre: Skipped 1 previous similar message [ 700.535153] Lustre: 25303:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 700.656301] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 707.957505] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 708.789206] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 708.807484] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 708.875319] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 713.719132] Lustre: 24307:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 713.728059] Lustre: 24307:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 713.737847] Lustre: 24307:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 713.750100] Lustre: 24307:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 713.770024] Lustre: 24307:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 713.787418] Lustre: 24307:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 735.742461] Lustre: 24307:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 735.775644] Lustre: 24307:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 735.798609] Lustre: 24307:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 735.837486] Lustre: 24307:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 735.854912] Lustre: 24307:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 735.879613] Lustre: 24307:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 742.075970] Lustre: 24307:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 742.090351] Lustre: 24307:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 742.098645] Lustre: 24307:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 742.110629] Lustre: 24307:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 742.130185] Lustre: 24307:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 742.143909] Lustre: 24307:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 742.155496] Lustre: 24307:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 742.168445] Lustre: 24307:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 742.186469] Lustre: 24307:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 742.205647] Lustre: 24307:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 742.228845] Lustre: 24307:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 742.248422] Lustre: 24307:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 745.611124] Lustre: 24308:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 745.644709] Lustre: 24308:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 745.666847] Lustre: 24308:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 745.681876] Lustre: 24308:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 745.705916] Lustre: 24308:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 745.729349] Lustre: 24308:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 745.748296] Lustre: 24308:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 745.763646] Lustre: 24308:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 745.787941] Lustre: 24308:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 745.807775] Lustre: 24308:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 745.822555] Lustre: 24308:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 745.830271] Lustre: 24308:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 754.325747] Lustre: 24307:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 754.331123] Lustre: 24307:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 754.333882] Lustre: 24307:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 754.339237] Lustre: 24307:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 754.343580] Lustre: 24307:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 754.348765] Lustre: 24307:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 754.353818] Lustre: 24307:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 754.359478] Lustre: 24307:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 754.365379] Lustre: 24307:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 754.369904] Lustre: 24307:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 754.374658] Lustre: 24307:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 754.378850] Lustre: 24307:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 762.969558] Lustre: 24307:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 762.980454] Lustre: 24307:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 762.986951] Lustre: 24307:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 762.992019] Lustre: 24307:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 762.999435] Lustre: 24307:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 763.006939] Lustre: 24307:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 763.017460] Lustre: 24307:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 763.022020] Lustre: 24307:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 763.026429] Lustre: 24307:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 763.030087] Lustre: 24307:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 763.033963] Lustre: 24307:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 763.037383] Lustre: 24307:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 816.100036] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 816.119523] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 820.785213] Lustre: server umount lustre-OST07c6 complete [ 830.945534] Lustre: server umount lustre-MDT0000 complete [ 852.014299] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 855.562051] Key type lgssc unregistered [ 855.949799] LNet: 27824:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 855.966553] LNetError: 27824:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 857.000963] LNet: Removed LNI 192.168.202.154@tcp [ 858.070626] Key type .llcrypt unregistered [ 858.077467] Key type ._llcrypt unregistered [ 884.580448] Key type ._llcrypt registered [ 884.584330] Key type .llcrypt registered [ 884.719474] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 886.669750] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 886.963695] alg: No test for adler32 (adler32-zlib) [ 888.285700] Lustre: Lustre: Build Version: 2.17.53_64_gab2e0dd [ 888.598725] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 890.423235] Key type lgssc registered [ 892.070071] Lustre: Echo OBD driver; http://www.lustre.org/ [ 903.039335] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 903.061744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 904.844168] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 910.599457] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 914.402374] LustreError: 29083:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 914.424533] LustreError: 29083:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 919.530793] LustreError: 29082:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 920.668771] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 921.225718] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 926.925514] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 934.338355] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 939.084968] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 947.209613] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 947.620789] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 949.696170] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 954.518229] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 961.166692] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 985.574841] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 985.582814] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 985.600116] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 986.595735] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 986.599256] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 986.608567] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 986.646675] Lustre: Skipped 1 previous similar message [ 988.902018] Lustre: server umount lustre-OST0000 complete [ 992.738063] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 992.748219] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 992.772378] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 996.835019] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 996.841402] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1000.914516] Lustre: server umount lustre-MDT0000 complete [ 1001.953855] LustreError: 30698:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1001.971043] LustreError: 30698:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1004.474525] LustreError: 29063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781229078 with bad export cookie 12301776102089692240 [ 1004.476167] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1004.481408] LustreError: 29063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1004.932985] Lustre: server umount lustre-MDT0001 complete [ 1016.052137] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 1020.083855] Key type lgssc unregistered [ 1020.472877] LNet: 32466:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1020.484820] LNetError: 32466:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1020.501785] LNet: Removed LNI 192.168.202.154@tcp [ 1021.700323] Key type .llcrypt unregistered [ 1021.705817] Key type ._llcrypt unregistered [ 1046.138641] Key type ._llcrypt registered [ 1046.141810] Key type .llcrypt registered [ 1046.343923] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 1048.335266] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1048.419423] alg: No test for adler32 (adler32-zlib) [ 1049.560621] Lustre: Lustre: Build Version: 2.17.53_64_gab2e0dd [ 1049.875628] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 1051.615835] Key type lgssc registered [ 1053.189479] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1062.890630] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1062.904787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1064.431607] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1069.449519] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 1074.669398] LustreError: 33721:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1074.705301] LustreError: 33721:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1079.443888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1079.785464] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1079.934067] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1085.394845] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 1092.611951] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1097.698125] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1106.247769] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1106.555296] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1109.617948] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1114.250728] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 1121.485814] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1140.711324] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1140.726293] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1140.729760] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1140.755802] Lustre: Skipped 1 previous similar message [ 1142.765106] Lustre: server umount lustre-OST0000 complete [ 1150.944760] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1150.947397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1150.968962] Lustre: Skipped 2 previous similar messages [ 1153.259555] Lustre: server umount lustre-MDT0000 complete [ 1156.064807] LustreError: 33721:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1156.086026] LustreError: 33721:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1158.089557] LustreError: 33703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781229232 with bad export cookie 1516486878621029993 [ 1158.103828] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1158.108682] LustreError: 33703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1158.632157] Lustre: server umount lustre-MDT0001 complete [ 1171.664374] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 1174.607463] Key type lgssc unregistered [ 1174.864566] LNet: 36921:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1174.871556] LNetError: 36921:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1175.921132] LNet: Removed LNI 192.168.202.154@tcp [ 1177.151483] Key type .llcrypt unregistered [ 1177.157705] Key type ._llcrypt unregistered [ 1209.219805] Key type ._llcrypt registered [ 1209.223593] Key type .llcrypt registered [ 1209.401088] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 1210.945776] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1211.218918] alg: No test for adler32 (adler32-zlib) [ 1212.537883] Lustre: Lustre: Build Version: 2.17.53_64_gab2e0dd [ 1212.987935] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 1214.855164] Key type lgssc registered [ 1216.444501] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1225.123378] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1225.173548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1226.784769] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1232.539519] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 1236.960953] LustreError: 38321:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1236.978868] LustreError: 38321:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1241.346418] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1241.924407] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1246.994874] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 1253.600575] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1257.662412] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1264.907828] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1265.365487] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1271.562547] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1271.612963] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 1278.056113] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1285.106068] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.54@tcp (stopping) [ 1287.139178] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1287.160604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1287.169822] Lustre: Skipped 2 previous similar messages [ 1290.214327] LustreError: 39040:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1290.344106] Lustre: server umount lustre-MDT0000 complete [ 1292.262258] LustreError: 38317:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1292.268076] LustreError: 38317:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1294.121286] LustreError: 38303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781229368 with bad export cookie 18251370301932399483 [ 1294.142984] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1295.316862] LustreError: 38318:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1295.349496] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.54@tcp (stopping) [ 1297.377744] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1297.388679] Lustre: Skipped 2 previous similar messages [ 1297.406887] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1299.457380] LustreError: 39040:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1299.480038] LustreError: 39040:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1300.472488] Lustre: server umount lustre-MDT0001 complete [ 1322.545900] Lustre: server umount lustre-OST0000 complete [ 1340.968560] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 1344.618924] Key type lgssc unregistered [ 1344.875233] LNet: 41835:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1344.888658] LNetError: 41835:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1344.906629] LNet: Removed LNI 192.168.202.154@tcp [ 1345.871647] Key type .llcrypt unregistered [ 1345.874434] Key type ._llcrypt unregistered [ 1371.909513] Key type ._llcrypt registered [ 1371.911679] Key type .llcrypt registered [ 1372.071620] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 1373.269728] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1373.437252] alg: No test for adler32 (adler32-zlib) [ 1374.648158] Lustre: Lustre: Build Version: 2.17.53_64_gab2e0dd [ 1374.908617] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 1376.649072] Key type lgssc registered [ 1378.161717] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1386.687596] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1386.734658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1388.306452] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1392.581381] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 1398.761941] LustreError: 43239:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1398.788913] LustreError: 43239:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1402.876748] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1403.270335] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1407.581483] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 1414.087585] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1418.806791] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1426.496437] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1427.011066] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1433.157710] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1434.038563] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 1440.828899] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1447.934654] Lustre: lustre-OST0000: Not available for connect from 192.168.202.54@tcp (stopping) [ 1448.422588] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1448.435316] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1448.442569] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1448.451413] Lustre: Skipped 1 previous similar message [ 1453.019536] Lustre: lustre-OST0000: Not available for connect from 192.168.202.54@tcp (stopping) [ 1453.026473] Lustre: Skipped 1 previous similar message [ 1453.291620] Lustre: server umount lustre-OST0000 complete [ 1474.019326] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1474.029961] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1474.034048] Lustre: Skipped 1 previous similar message [ 1474.058108] Lustre: Skipped 1 previous similar message [ 1479.136754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1479.143106] Lustre: Skipped 1 previous similar message [ 1486.303719] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1486.445506] Lustre: server umount lustre-MDT0000 complete [ 1488.365187] LustreError: 44223:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1488.392457] LustreError: 44223:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1490.102896] LustreError: 43219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781229564 with bad export cookie 10388759398589956843 [ 1490.117146] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1496.559043] Lustre: server umount lustre-MDT0001 complete [ 1509.179448] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 1512.180721] Key type lgssc unregistered [ 1512.615553] LNet: 46592:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1512.629512] LNetError: 46592:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1512.652994] LNet: Removed LNI 192.168.202.154@tcp [ 1513.614994] Key type .llcrypt unregistered [ 1513.617340] Key type ._llcrypt unregistered [ 1538.734315] Key type ._llcrypt registered [ 1538.737555] Key type .llcrypt registered [ 1538.901937] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 1540.570304] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1540.751611] alg: No test for adler32 (adler32-zlib) [ 1541.873916] Lustre: Lustre: Build Version: 2.17.53_64_gab2e0dd [ 1542.103741] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 1543.831177] Key type lgssc registered [ 1545.134849] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1553.464246] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1553.472483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1554.977508] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1560.780621] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 1564.133804] LustreError: 47994:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1564.167537] LustreError: 47994:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1569.279254] LustreError: 47995:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1571.332866] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1571.987708] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1577.490559] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 1584.303709] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1589.012367] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1597.027924] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1597.609875] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1599.683685] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1604.515627] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 1611.089093] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1621.131473] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1625.990912] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1638.199144] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-06-1122h01m49s [ 1639.992993] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1643.488870] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1643.499545] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1643.528498] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1644.563239] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.54@tcp (stopping) [ 1644.572050] Lustre: Skipped 2 previous similar messages [ 1645.656804] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.54@tcp (stopping) [ 1645.666420] Lustre: Skipped 3 previous similar messages [ 1648.773278] Lustre: server umount lustre-MDT0000 complete [ 1651.169195] LustreError: 47994:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1651.196822] LustreError: 47994:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1652.755459] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1657.315608] LustreError: 48706:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1657.351982] LustreError: 48706:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1660.161477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1660.303188] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1660.680534] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1660.783943] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1661.678310] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1665.838551] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 1669.655975] Lustre: DEBUG MARKER: Wait for df (36058) ... [ 1671.428524] Lustre: DEBUG MARKER: done [ 1676.260885] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1676.273596] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1676.295335] Lustre: Skipped 2 previous similar messages [ 1676.300053] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1677.285681] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1682.195192] Lustre: server umount lustre-OST0000 complete [ 1687.531989] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1687.536329] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1687.546840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1687.555385] Lustre: Skipped 3 previous similar messages [ 1694.700314] LustreError: 47994:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1694.714852] LustreError: 47994:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1694.738839] Lustre: server umount lustre-MDT0000 complete [ 1698.459408] LustreError: 47975:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781229772 with bad export cookie 12649375462481199359 [ 1698.473444] LustreError: 47975:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1698.479126] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1698.953476] Lustre: server umount lustre-MDT0001 complete [ 1711.982317] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 1715.417953] Key type lgssc unregistered [ 1715.745770] LNet: 52562:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1715.757562] LNetError: 52562:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1715.773646] LNet: Removed LNI 192.168.202.154@tcp [ 1716.617247] Key type .llcrypt unregistered [ 1716.619691] Key type ._llcrypt unregistered [ 1763.445973] Key type ._llcrypt registered [ 1763.447667] Key type .llcrypt registered [ 1763.554461] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 1765.246633] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1765.308122] alg: No test for adler32 (adler32-zlib) [ 1766.354293] Lustre: Lustre: Build Version: 2.17.53_64_gab2e0dd [ 1766.560167] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 1768.287873] Key type lgssc registered [ 1769.857236] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1779.009079] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1779.035140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1780.457402] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1780.489531] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1780.685284] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1786.045189] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 1795.673421] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1795.836419] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1795.871321] Lustre: 54758:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1796.187897] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1801.125175] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 1807.474572] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1813.247360] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1821.960836] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1822.231311] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1822.531739] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1830.696981] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 1832.674926] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1838.553280] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1845.505572] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-06-1122h05m17s [ 1847.758596] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1856.754913] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1861.535311] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1861.703244] LustreError: 54765:0:(mdt_handler.c:2347:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1882.120572] Lustre: lustre-MDT0000: Client b5df94ed-9ce3-43a2-b7d8-c1e4c905c8e2 (at 192.168.202.54@tcp) reconnecting [ 1901.711543] LustreError: 54765:0:(mdt_handler.c:2347:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1904.516539] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1907.587941] Lustre: DEBUG MARKER: done [ 1914.855636] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1914.858577] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1914.860991] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1914.868064] LustreError: Skipped 1 previous similar message [ 1914.920985] Lustre: Skipped 1 previous similar message [ 1919.971597] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1919.984213] Lustre: Skipped 2 previous similar messages [ 1921.069691] Lustre: server umount lustre-OST0000 complete [ 1929.194194] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1929.202816] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1929.227463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1931.909868] Lustre: server umount lustre-MDT0000 complete [ 1935.968670] LustreError: 54747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781230009 with bad export cookie 5518135522304164165 [ 1935.982927] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1935.988401] LustreError: 54747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1936.791026] Lustre: server umount lustre-MDT0001 complete [ 1950.601171] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 1953.805142] Key type lgssc unregistered [ 1954.130691] LNet: 58731:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1954.136216] LNetError: 58731:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1954.151403] LNet: Removed LNI 192.168.202.154@tcp [ 1955.056473] Key type .llcrypt unregistered [ 1955.058097] Key type ._llcrypt unregistered [ 2001.454514] Key type ._llcrypt registered [ 2001.456150] Key type .llcrypt registered [ 2001.595291] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 2003.628722] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2003.668069] alg: No test for adler32 (adler32-zlib) [ 2004.858391] Lustre: Lustre: Build Version: 2.17.53_64_gab2e0dd [ 2005.174223] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 2006.999238] Key type lgssc registered [ 2008.246718] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2011.477814] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2011.494692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2012.959389] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2013.007903] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2013.318359] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2014.475901] Lustre: Failing over lustre-MDT0000 [ 2014.891382] Lustre: server umount lustre-MDT0000 complete [ 2025.999568] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 22:08:17 (1781230097) [ 2035.737056] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2036.664926] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2043.070097] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 2053.628582] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2053.703275] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2053.729484] Lustre: 61517:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2054.014685] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2059.613082] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 2067.036844] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2071.967077] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2080.020589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2080.251671] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2080.503291] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2087.268658] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 2088.621716] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2093.449631] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2099.978062] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2109.409681] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2109.414576] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2109.437570] Lustre: Skipped 1 previous similar message [ 2109.460089] Lustre: Skipped 2 previous similar messages [ 2113.703339] Lustre: server umount lustre-MDT0000 complete [ 2117.298514] LustreError: 61507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781230191 with bad export cookie 2465920163720358214 [ 2117.306605] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2117.309598] LustreError: 61507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2117.574365] Lustre: server umount lustre-MDT0001 complete [ 2121.699592] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2127.511937] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2135.008395] Lustre: 60438:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781230193/real 1781230193] req@0000000085e6fdc6 x1867755111358464/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781230209 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2135.083210] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2135.099565] Lustre: Skipped 1 previous similar message [ 2137.197475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2138.080291] Lustre: 60439:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781230196/real 1781230196] req@0000000031a982d1 x1867755111358592/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781230212 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2140.127353] Lustre: 60440:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781230198/real 1781230198] req@0000000084cad09f x1867755111358720/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781230214 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2142.305513] LustreError: 60436:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000084cad09f x1867755111359488/t0(0) o250->MGC192.168.202.154@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2142.709098] LustreError: 63138:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2142.743571] LustreError: 63138:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2142.848340] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2143.007234] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2143.775127] Lustre: 60437:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781230201/real 1781230201] req@0000000029fcbc73 x1867755111358976/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781230217 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2144.877673] LustreError: 63139:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2145.899273] LustreError: 64946:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2145.923869] LustreError: 64946:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2147.885103] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 2148.959539] Lustre: 60438:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781230206/real 1781230206] req@00000000023594f8 x1867755111359104/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781230222 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2151.405325] LustreError: 63138:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2151.421471] LustreError: 63138:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2156.517923] LustreError: 64945:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2156.548185] LustreError: 64945:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2158.529463] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2159.200136] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2164.223615] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2164.536667] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 2165.307184] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2171.575862] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2176.486408] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2184.672443] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2184.691533] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2184.701991] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2185.715952] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2185.732064] Lustre: Skipped 2 previous similar messages [ 2190.834330] LustreError: 65853:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2191.014658] Lustre: server umount lustre-MDT0000 complete [ 2195.085155] LustreError: 61507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781230269 with bad export cookie 2465920163720360496 [ 2195.094142] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2195.100354] LustreError: 61507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2195.529790] Lustre: server umount lustre-MDT0001 complete [ 2202.520149] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2211.300861] Lustre: 60440:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781230269/real 1781230269] req@000000006a654003 x1867755111387648/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781230285 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2211.352580] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2211.381985] Lustre: Skipped 2 previous similar messages [ 2213.392643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2220.640308] LustreError: 60436:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000034f6854e x1867755111388544/t0(0) o250->MGC192.168.202.154@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2221.110350] LustreError: 63139:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2221.138907] LustreError: 63139:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2221.284717] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2221.376976] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2228.058667] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 2238.862727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2240.565864] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2244.320812] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 2250.437114] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2254.882917] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2261.449535] Lustre: server umount lustre-MDT0000 complete [ 2265.059394] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2265.070536] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2265.084816] LustreError: 65662:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2265.099584] LustreError: 65662:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 2265.229063] LustreError: 61508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781230339 with bad export cookie 2465920163720361875 [ 2265.240249] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2265.243954] LustreError: 61508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2265.539638] Lustre: server umount lustre-MDT0001 complete [ 2272.054367] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2275.754594] Lustre: DEBUG MARKER: files compared the same [ 2282.976336] Lustre: 60439:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781230340/real 1781230340] req@000000001caefdcc x1867755111413504/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781230356 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2283.014267] Lustre: 60439:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 2283.020215] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2287.517711] Lustre: server umount lustre-OST0000 complete [ 2305.969291] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 2309.341705] Key type lgssc unregistered [ 2309.682397] LNet: 70434:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2309.692621] LNetError: 70434:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2310.782496] LNet: Removed LNI 192.168.202.154@tcp [ 2312.225958] Key type .llcrypt unregistered [ 2312.228959] Key type ._llcrypt unregistered [ 2336.430294] Key type ._llcrypt registered [ 2336.438332] Key type .llcrypt registered [ 2336.554632] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 2338.164080] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2338.213757] alg: No test for adler32 (adler32-zlib) [ 2339.254842] Lustre: Lustre: Build Version: 2.17.53_64_gab2e0dd [ 2339.464952] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 2341.127239] Key type lgssc registered [ 2342.298902] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2349.988271] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2350.020321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2351.516934] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2355.998243] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug +malloc all [ 2361.831856] LustreError: 71834:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2361.865756] LustreError: 71834:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2365.825731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2366.248349] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2371.678460] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug +malloc all [ 2377.999269] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2381.449584] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2388.318350] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2388.638353] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2394.451678] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug +malloc all [ 2396.851924] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2400.066681] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2412.516031] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2412.516725] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2412.517601] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2412.534058] Lustre: Skipped 1 previous similar message [ 2412.547669] LustreError: Skipped 1 previous similar message [ 2415.316757] Lustre: server umount lustre-OST0000 complete [ 2422.752021] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2422.752746] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2422.770432] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2422.798648] Lustre: Skipped 2 previous similar messages [ 2425.004993] Lustre: server umount lustre-MDT0000 complete [ 2427.871627] LustreError: 71834:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2427.895431] LustreError: 71834:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2429.375192] LustreError: 71814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781230503 with bad export cookie 13247674561788023481 [ 2429.383951] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2429.390816] LustreError: 71814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2429.716650] Lustre: server umount lustre-MDT0001 complete [ 2442.302559] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 2445.749611] Key type lgssc unregistered [ 2445.984215] LNet: 75010:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2445.991557] LNetError: 75010:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2446.020112] LNet: Removed LNI 192.168.202.154@tcp [ 2446.782617] Key type .llcrypt unregistered [ 2446.790109] Key type ._llcrypt unregistered [ 2473.395814] Key type ._llcrypt registered [ 2473.397635] Key type .llcrypt registered [ 2473.502314] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 2474.507337] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2474.583845] alg: No test for adler32 (adler32-zlib) [ 2475.684525] Lustre: Lustre: Build Version: 2.17.53_64_gab2e0dd [ 2475.871875] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 2477.576643] Key type lgssc registered [ 2478.713777] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2486.967831] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2486.995511] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2502.495342] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2507.615516] LustreError: 76382:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.154@tcp: failed processing log, type 4: rc = -110 [ 2534.304960] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2540.758032] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 2546.247120] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2553.676694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2553.781585] Lustre: *** cfs_fail_loc=706, val=0*** [ 2554.260784] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2554.372676] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2559.045358] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 2559.465212] LustreError: 76407:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2559.488312] LustreError: 76407:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2564.585186] LustreError: 76408:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2567.463851] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2567.829649] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2572.605975] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 2577.434905] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2581.467227] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2585.065165] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2585.069131] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2585.096653] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2585.571608] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2590.716257] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2590.729128] Lustre: Skipped 1 previous similar message [ 2595.814676] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2595.827513] Lustre: Skipped 2 previous similar messages [ 2598.367566] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2598.614497] Lustre: server umount lustre-OST0000 complete [ 2603.495233] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2603.504595] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2603.516816] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2606.049435] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2609.096583] Lustre: server umount lustre-MDT0000 complete [ 2611.168800] LustreError: 77354:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2611.185567] LustreError: 77354:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2612.677435] LustreError: 76388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781230686 with bad export cookie 17584162738272384205 [ 2612.681269] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2612.685062] LustreError: 76388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2613.067359] Lustre: server umount lustre-MDT0001 complete [ 2625.378364] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 2628.573748] Key type lgssc unregistered [ 2628.877915] LNet: 79642:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2628.887574] LNetError: 79642:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2628.904030] LNet: Removed LNI 192.168.202.154@tcp [ 2629.624616] Key type .llcrypt unregistered [ 2629.629341] Key type ._llcrypt unregistered [ 2657.158562] Key type ._llcrypt registered [ 2657.161174] Key type .llcrypt registered [ 2657.247663] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 2658.204155] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2658.306048] alg: No test for adler32 (adler32-zlib) [ 2659.338949] Lustre: Lustre: Build Version: 2.17.53_64_gab2e0dd [ 2659.562904] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 2661.279142] Key type lgssc registered [ 2662.451064] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2670.734333] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2670.760122] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2675.672974] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 2684.715033] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2685.716235] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2690.847043] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 2691.048933] LustreError: 81715:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2691.069965] LustreError: 81715:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2695.140120] LustreError: 81714:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2695.181197] LustreError: 81714:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2700.264665] LustreError: 81718:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2700.282681] LustreError: 81718:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2701.503150] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2701.839926] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2706.922756] LustreError: 81715:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2706.946259] LustreError: 81715:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2709.073912] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 2711.013608] LustreError: 81714:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2711.021554] LustreError: 81714:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2719.168532] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2719.192672] Lustre: MGS: Client df6a115a-71b1-4030-8eaf-79f610009a18 (at 0@lo) reconnecting [ 2719.232527] LustreError: 81729:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2719.466441] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2719.568881] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2724.187577] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 2740.192880] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2740.202600] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2740.217295] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2740.230441] Lustre: Skipped 1 previous similar message [ 2740.252864] Lustre: Skipped 1 previous similar message [ 2744.758482] Lustre: server umount lustre-OST0000 complete [ 2750.438818] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2750.440950] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2750.444607] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2750.452775] Lustre: Skipped 1 previous similar message [ 2750.460100] Lustre: Skipped 1 previous similar message [ 2755.553019] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2755.560273] Lustre: Skipped 1 previous similar message [ 2760.674163] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2760.682056] Lustre: Skipped 1 previous similar message [ 2763.232863] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2763.372988] Lustre: server umount lustre-MDT0000 complete [ 2765.793743] LustreError: 81719:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2765.809243] LustreError: 81719:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2767.598672] Lustre: server umount lustre-MDT0001 complete [ 2771.656192] Lustre: server umount MGS complete [ 2784.153888] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 22:20:56 (1781230856) [ 2814.237830] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [ 2823.481375] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 2856.763201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2861.972746] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 2873.736422] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2874.012127] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2874.049424] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2874.116510] Lustre: lustre-MDT0001: new disk, initializing [ 2874.231293] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2874.262755] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2874.265597] Lustre: 88786:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2874.278323] LustreError: 88786:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2879.948765] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 2891.601617] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2891.819123] Lustre: lustre-OST0000: new disk, initializing [ 2891.822429] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2891.832340] Lustre: 89535:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2891.960241] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2897.270576] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 2899.619081] Lustre: 89936:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@00000000b7add713 x1867755797128064/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1781230989 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2905.941572] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2913.729773] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2913.747819] Lustre: MGS: Client 218025fa-bae3-4715-a72f-cf1ada62daf9 (at 0@lo) reconnecting [ 2913.813768] Lustre: 90468:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2913.829587] LustreError: 88772:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2913.861071] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2913.943626] Lustre: lustre-MDT0000: new disk, initializing [ 2914.119424] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2914.152711] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2914.206112] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2915.234711] Lustre: 90492:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@000000004a5350c8 x1867755797134976/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1781231005 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2918.899618] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 2920.418065] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2924.393084] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2924.423386] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2924.496478] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2950.117802] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2950.127112] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2950.140651] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2955.499686] Lustre: server umount lustre-OST0000 complete [ 2961.891584] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2961.894977] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2961.914624] Lustre: Skipped 1 previous similar message [ 2961.922834] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2961.932779] Lustre: Skipped 3 previous similar messages [ 2965.741206] Lustre: server umount lustre-MDT0000 complete [ 2967.012335] LustreError: 90819:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2967.027457] LustreError: 90819:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2976.209548] Lustre: server umount lustre-MDT0001 complete [ 2979.712499] Lustre: server umount MGS complete [ 2991.531368] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 22:24:24 (1781231064) [ 3013.609517] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 3016.554226] Key type lgssc unregistered [ 3016.801312] LNet: 92954:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3016.806175] LNetError: 92954:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3016.818626] LNet: Removed LNI 192.168.202.154@tcp [ 3017.441615] Key type .llcrypt unregistered [ 3017.443711] Key type ._llcrypt unregistered [ 3036.776907] Key type ._llcrypt registered [ 3036.781883] Key type .llcrypt registered [ 3036.904669] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 3038.099817] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3038.297091] alg: No test for adler32 (adler32-zlib) [ 3039.396538] Lustre: Lustre: Build Version: 2.17.53_64_gab2e0dd [ 3039.700201] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 3041.400914] Key type lgssc registered [ 3043.102898] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3051.727795] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3051.743778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3053.302274] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3058.363957] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 3063.785788] LustreError: 94163:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3063.820552] LustreError: 94163:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3067.730232] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3068.138622] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3072.938112] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 3078.469834] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3082.419469] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3090.302048] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3090.702251] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3097.414407] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 3098.933208] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3103.736262] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3114.467285] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3114.475899] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3114.485047] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3114.485940] Lustre: Skipped 1 previous similar message [ 3119.639492] Lustre: server umount lustre-OST0000 complete [ 3124.705866] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3124.709958] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3124.711347] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3124.713838] LustreError: Skipped 1 previous similar message [ 3124.732802] Lustre: Skipped 2 previous similar messages [ 3129.714363] Lustre: server umount lustre-MDT0000 complete [ 3129.824456] LustreError: 94157:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3129.844460] LustreError: 94157:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3133.641199] LustreError: 94144:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781231207 with bad export cookie 13907320395780577541 [ 3133.656740] LustreError: 94144:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3133.660459] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3134.291805] Lustre: server umount lustre-MDT0001 complete [ 3146.874712] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 3150.161897] Key type lgssc unregistered [ 3150.527136] LNet: 97343:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3150.536092] LNetError: 97343:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3150.567733] LNet: Removed LNI 192.168.202.154@tcp [ 3151.771654] Key type .llcrypt unregistered [ 3151.775215] Key type ._llcrypt unregistered [ 3170.680646] Key type ._llcrypt registered [ 3170.685284] Key type .llcrypt registered [ 3170.880771] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 3172.127833] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3172.244154] alg: No test for adler32 (adler32-zlib) [ 3173.474075] Lustre: Lustre: Build Version: 2.17.53_64_gab2e0dd [ 3173.793754] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 3175.447316] Key type lgssc registered [ 3177.277562] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3184.873570] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3184.877858] LustreError: 98410:0:(tgt_mount.c:2522:server_fill_super()) cfs_race id 716 sleeping [ 3184.894539] LustreError: 98411:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 waking [ 3184.899159] LustreError: 98410:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 awake: rc=4983 [ 3184.901670] LustreError: 98410:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3184.911524] LustreError: 98410:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3184.918572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3186.641795] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3195.431312] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3195.627138] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3200.554938] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 3205.842867] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing lsmod [ 3208.876871] LustreError: 99577:0:(tgt_mount.c:2522:server_fill_super()) cfs_race id 716 sleeping [ 3208.950494] LustreError: 99584:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 waking [ 3208.963659] LustreError: 99577:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 awake: rc=4921 [ 3208.974778] LustreError: 99577:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3208.985688] LustreError: 99577:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3209.000234] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3209.403872] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3211.440628] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3216.355219] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3216.357852] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3216.381499] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3216.872067] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3216.883400] Lustre: Skipped 1 previous similar message [ 3216.886634] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3216.895519] Lustre: Skipped 1 previous similar message [ 3219.143225] Lustre: server umount lustre-MDT0000 complete [ 3220.971472] LustreError: 98931:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3221.009155] LustreError: 98931:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3222.917508] LustreError: 98424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781231296 with bad export cookie 11183572430087242537 [ 3222.921657] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3222.927159] LustreError: 98424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3223.064207] LustreError: 98879:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000005e30f304 x1867756336075008/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3223.267510] Lustre: server umount lustre-MDT0001 complete [ 3237.727890] Lustre: server umount lustre-OST0000 complete [ 3246.799506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3247.312609] LustreError: 100646:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3247.410076] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3252.364901] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 3252.709783] LustreError: 100647:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3257.839367] LustreError: 100646:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3261.311806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3261.605807] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3265.916568] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 3273.054727] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3277.067303] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3285.357023] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3285.826459] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3292.027157] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97) [ 3292.806317] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 3299.447960] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3307.489612] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3307.499045] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3307.505922] Lustre: Skipped 1 previous similar message [ 3307.518282] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3307.518853] Lustre: Skipped 1 previous similar message [ 3308.890449] Lustre: server umount lustre-OST0000 complete [ 3317.733607] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3317.736940] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3317.754918] Lustre: Skipped 1 previous similar message [ 3317.765030] Lustre: Skipped 1 previous similar message [ 3319.695161] Lustre: server umount lustre-MDT0000 complete [ 3322.854958] LustreError: 100642:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3322.877709] LustreError: 100642:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3324.076215] LustreError: 100627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781231398 with bad export cookie 11183572430087243454 [ 3324.091599] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3324.505982] Lustre: server umount lustre-MDT0001 complete [ 3335.528906] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 3338.279272] Key type lgssc unregistered [ 3338.515515] LNet: 103820:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3338.521946] LNetError: 103820:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3338.546523] LNet: Removed LNI 192.168.202.154@tcp [ 3339.293414] Key type .llcrypt unregistered [ 3339.295136] Key type ._llcrypt unregistered [ 3362.950448] Key type ._llcrypt registered [ 3362.952276] Key type .llcrypt registered [ 3363.056265] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 3364.237694] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3364.379319] alg: No test for adler32 (adler32-zlib) [ 3365.355385] Lustre: Lustre: Build Version: 2.17.53_64_gab2e0dd [ 3365.540912] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 3367.191616] Key type lgssc registered [ 3368.169743] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3376.143695] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3376.183667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3377.659797] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3382.565620] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 3387.881691] LustreError: 105220:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3387.895091] LustreError: 105220:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3391.569689] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3392.012201] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3397.717783] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 3404.410223] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3409.663789] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3417.372891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3418.239741] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3422.347033] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3426.823383] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 3434.837679] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3441.924615] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3445.819694] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3452.391445] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3452.419334] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3452.420281] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3452.449410] Lustre: Skipped 1 previous similar message [ 3456.844272] Lustre: server umount lustre-OST0000 complete [ 3462.631842] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3462.651696] Lustre: Skipped 1 previous similar message [ 3462.656822] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3462.665026] Lustre: Skipped 1 previous similar message [ 3467.762557] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3467.776521] Lustre: Skipped 1 previous similar message [ 3468.027632] Lustre: server umount lustre-MDT0000 complete [ 3472.591365] LustreError: 105201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781231546 with bad export cookie 7072268821139953427 [ 3472.598397] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3472.978247] Lustre: server umount lustre-MDT0001 complete [ 3485.516262] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 3489.187709] Key type lgssc unregistered [ 3489.426707] LNet: 108504:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3489.430246] LNetError: 108504:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3489.443766] LNet: Removed LNI 192.168.202.154@tcp [ 3490.221987] Key type .llcrypt unregistered [ 3490.223039] Key type ._llcrypt unregistered [ 3507.885988] Key type ._llcrypt registered [ 3507.888786] Key type .llcrypt registered [ 3508.023048] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 3509.845192] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3509.877791] alg: No test for adler32 (adler32-zlib) [ 3511.125434] Lustre: Lustre: Build Version: 2.17.53_64_gab2e0dd [ 3511.546193] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 3513.304234] Key type lgssc registered [ 3514.653369] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3523.423209] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3523.440384] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3525.039967] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3529.951940] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 3535.375956] LustreError: 109714:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3535.392283] LustreError: 109714:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3538.965836] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3539.393732] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3545.659929] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 3553.164843] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3557.781720] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3565.448361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3565.887195] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3571.002309] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3571.716625] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 3578.249716] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3593.199787] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3593.205561] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3593.235832] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3594.722829] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3594.723139] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3594.730960] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3599.282570] Lustre: server umount lustre-OST0000 complete [ 3604.981524] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3604.982724] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3604.996180] Lustre: Skipped 1 previous similar message [ 3605.020695] Lustre: Skipped 2 previous similar messages [ 3609.915477] Lustre: server umount lustre-MDT0000 complete [ 3610.083899] LustreError: 111865:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3610.099458] LustreError: 111865:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3614.176550] LustreError: 109714:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3614.196259] LustreError: 109714:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3614.314641] LustreError: 109694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781231688 with bad export cookie 11735174643348151816 [ 3614.316510] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3614.323829] LustreError: 109694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3614.891671] Lustre: server umount lustre-MDT0001 complete [ 3629.380724] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 3632.371549] Key type lgssc unregistered [ 3632.719316] LNet: 112912:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3632.735263] LNetError: 112912:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3632.767049] LNet: Removed LNI 192.168.202.154@tcp [ 3633.653713] Key type .llcrypt unregistered [ 3633.655613] Key type ._llcrypt unregistered [ 3659.469495] Key type ._llcrypt registered [ 3659.472365] Key type .llcrypt registered [ 3659.609646] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 3660.524750] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3660.582155] alg: No test for adler32 (adler32-zlib) [ 3661.722249] Lustre: Lustre: Build Version: 2.17.53_64_gab2e0dd [ 3661.936660] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 3663.599251] Key type lgssc registered [ 3664.933858] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3672.948158] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3672.987588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3674.675599] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3679.820156] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 3684.841391] LustreError: 114310:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3684.871436] LustreError: 114310:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3689.562875] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3689.909343] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3694.682538] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 3701.929034] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3707.097230] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3715.321240] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3715.707551] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3719.816522] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 3721.627854] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 3728.139547] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3739.407097] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3743.889609] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3743.894302] Lustre: Skipped 2 previous similar messages [ 3745.910916] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3745.949989] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3749.060500] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3749.066209] Lustre: Skipped 2 previous similar messages [ 3752.556600] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3752.561500] Lustre: Skipped 1 previous similar message [ 3761.198547] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.54@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3761.206454] Lustre: Skipped 2 previous similar messages [ 3768.434697] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.54@tcp 192.168.0.[2,10]@tcp [ 3773.963294] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.54/24@tcp in log lustre-MDT0000 [ 3773.985507] Lustre: Skipped 2 previous similar messages [ 3778.776717] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.54/24@tcp [ 3778.786858] Lustre: Skipped 1 previous similar message [ 3795.941362] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3795.943385] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3795.954745] Lustre: Skipped 1 previous similar message [ 3798.536536] Lustre: server umount lustre-OST0000 complete [ 3806.192176] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3806.210760] Lustre: Skipped 1 previous similar message [ 3806.222217] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3806.233463] Lustre: Skipped 2 previous similar messages [ 3809.253398] Lustre: server umount lustre-MDT0000 complete [ 3811.305045] LustreError: 114310:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3811.325616] LustreError: 114310:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3813.942628] LustreError: 114291:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781231887 with bad export cookie 11124406912186896426 [ 3813.948296] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3813.957051] LustreError: 114291:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3814.430186] Lustre: server umount lustre-MDT0001 complete [ 3828.417976] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 3831.568563] Key type lgssc unregistered [ 3831.950682] LNet: 118902:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3831.956564] LNetError: 118902:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3831.971256] LNet: Removed LNI 192.168.202.154@tcp [ 3832.748674] Key type .llcrypt unregistered [ 3832.751666] Key type ._llcrypt unregistered [ 3868.798710] Key type ._llcrypt registered [ 3868.805357] Key type .llcrypt registered [ 3868.919163] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 3870.329760] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3870.396117] alg: No test for adler32 (adler32-zlib) [ 3871.546134] Lustre: Lustre: Build Version: 2.17.53_64_gab2e0dd [ 3871.849241] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 3873.519474] Key type lgssc registered [ 3874.999392] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3884.862693] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3884.881227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3886.521888] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3886.577762] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3886.579637] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.54/24@tcp [ 3891.920177] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 3896.806754] LustreError: 120451:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3896.841125] LustreError: 120451:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3901.925125] LustreError: 120450:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3903.151646] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3903.393600] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3903.434708] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.54/24@tcp [ 3903.518656] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3910.311812] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 3918.425088] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3923.566994] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3931.325884] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3931.796483] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3939.391850] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 3945.379755] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3956.352830] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:201 to 0x280000400:225) [ 3966.945567] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3966.949793] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3966.960660] Lustre: Skipped 1 previous similar message [ 3966.961619] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3966.968520] Lustre: Skipped 1 previous similar message [ 3972.126106] Lustre: server umount lustre-OST0000 complete [ 3977.186328] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3977.215571] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3977.227803] Lustre: Skipped 1 previous similar message [ 3981.991484] Lustre: server umount lustre-MDT0000 complete [ 3982.304460] LustreError: 120450:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3982.324884] LustreError: 120450:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3986.490261] LustreError: 122970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781232060 with bad export cookie 4657546077311215916 [ 3986.493258] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3986.498679] LustreError: 122970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3986.801595] Lustre: server umount lustre-MDT0001 complete [ 3999.353810] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 4003.230730] Key type lgssc unregistered [ 4003.702071] LNet: 123705:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4003.718895] LNetError: 123705:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4003.761327] LNet: Removed LNI 192.168.202.154@tcp [ 4005.373698] Key type .llcrypt unregistered [ 4005.377969] Key type ._llcrypt unregistered [ 4038.642600] Key type ._llcrypt registered [ 4038.644203] Key type .llcrypt registered [ 4038.707441] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [ 4053.080889] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 4054.226823] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4054.307705] alg: No test for adler32 (adler32-zlib) [ 4055.519907] Lustre: Lustre: Build Version: 2.17.53_64_gab2e0dd [ 4055.717804] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 4057.407202] Key type lgssc registered [ 4058.541898] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4093.117248] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4093.163031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4094.442391] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4094.475334] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4094.549754] Lustre: lustre-MDT0000: new disk, initializing [ 4094.699343] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4094.726995] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4098.678495] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 4109.885937] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4110.039731] Lustre: 127067:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4110.101300] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4110.112612] Lustre: Skipped 1 previous similar message [ 4110.184956] Lustre: lustre-MDT0001: new disk, initializing [ 4110.301740] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4110.346531] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4110.358936] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4115.635547] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 4124.391503] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4128.789325] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4137.821605] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4138.086332] Lustre: lustre-OST0000: new disk, initializing [ 4138.089584] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4138.094211] Lustre: 128812:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4138.185643] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4143.735975] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4143.747609] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4143.964362] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4145.686729] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 4154.574826] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4160.331877] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4160.532588] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4165.791589] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4166.018373] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4169.700414] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4169.705372] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4169.715051] LustreError: Skipped 1 previous similar message [ 4169.715853] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4169.735346] Lustre: Skipped 1 previous similar message [ 4169.761731] Lustre: Skipped 1 previous similar message [ 4174.481624] Lustre: server umount lustre-OST0000 complete [ 4179.942400] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4179.948928] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4179.965768] Lustre: Skipped 1 previous similar message [ 4179.977830] Lustre: Skipped 1 previous similar message [ 4185.059944] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4185.068326] Lustre: Skipped 1 previous similar message [ 4185.610293] Lustre: server umount lustre-MDT0000 complete [ 4190.300628] LustreError: 127061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781232264 with bad export cookie 117223937342865970 [ 4190.306785] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4190.314733] LustreError: 127061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4190.827144] Lustre: server umount lustre-MDT0001 complete [ 4200.630508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4201.100633] LustreError: 130884:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4201.119939] LustreError: 130884:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4201.189062] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4205.519763] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 4206.569680] LustreError: 130885:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4211.685274] LustreError: 130884:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4214.065304] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4214.324446] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4219.500757] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 4225.380526] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4228.859899] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4235.702372] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4236.062118] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4242.300650] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 4247.944449] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4251.694095] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4251.911204] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4256.813974] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4257.117129] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4266.592679] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4266.726570] Lustre: lustre-OST0001: new disk, initializing [ 4266.729913] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4266.734234] Lustre: 133881:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4266.809915] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4272.957282] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 4274.290194] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4274.296890] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4274.404994] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4281.506679] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4281.666469] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4285.333202] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4285.549254] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4289.966126] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff95d00baf2800.ost_server_uuid 50 [ 4291.527502] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff95d00baf2800.ost_server_uuid in FULL state after 0 sec [ 4300.256182] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4300.263682] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4300.274845] LustreError: Skipped 1 previous similar message [ 4300.278849] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4300.294989] Lustre: Skipped 1 previous similar message [ 4300.321571] Lustre: Skipped 1 previous similar message [ 4305.378218] LustreError: 132479:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4305.401745] LustreError: 132479:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4305.433275] Lustre: server umount lustre-OST0001 complete [ 4310.496828] LustreError: 132480:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4310.503619] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4310.539467] LustreError: 132480:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4310.571939] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4320.739772] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4320.747731] Lustre: Skipped 4 previous similar messages [ 4320.751186] LustreError: 132480:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4320.804922] LustreError: 132480:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4324.323450] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4324.598294] Lustre: server umount lustre-OST0000 complete [ 4330.976148] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4330.993167] Lustre: Skipped 2 previous similar messages [ 4334.452593] Lustre: server umount lustre-MDT0000 complete [ 4338.871019] LustreError: 130864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781232412 with bad export cookie 117223937342866852 [ 4338.873089] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4338.883330] LustreError: 130864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4339.520143] Lustre: server umount lustre-MDT0001 complete [ 4365.054684] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 4369.711776] Key type lgssc unregistered [ 4370.058879] LNet: 136732:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4370.075164] LNetError: 136732:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4370.107803] LNet: Removed LNI 192.168.202.154@tcp [ 4370.824746] Key type .llcrypt unregistered [ 4370.826331] Key type ._llcrypt unregistered [ 4419.097566] Key type ._llcrypt registered [ 4419.099452] Key type .llcrypt registered [ 4419.218636] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [ 4434.172301] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 4435.540750] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4435.550243] alg: No test for adler32 (adler32-zlib) [ 4436.706698] Lustre: Lustre: Build Version: 2.17.53_64_gab2e0dd [ 4436.966627] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 4438.639289] Key type lgssc registered [ 4439.736384] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4477.739239] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4477.768522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4479.190932] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4479.236733] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4479.355683] Lustre: lustre-MDT0000: new disk, initializing [ 4479.445337] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4479.464208] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4484.567859] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 4494.858434] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4495.061810] Lustre: 140870:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4495.131419] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4495.136618] Lustre: Skipped 1 previous similar message [ 4495.250741] Lustre: lustre-MDT0001: new disk, initializing [ 4495.373826] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4495.427585] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4495.438582] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4500.339441] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 4507.832703] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4512.084846] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4519.098737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4519.437215] Lustre: lustre-OST0000: new disk, initializing [ 4519.439908] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4519.448179] Lustre: 142614:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4519.520073] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4525.148930] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 4526.189313] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4526.200196] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4526.373604] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4533.082970] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4537.541674] Lustre: Failing over lustre-OST0000 [ 4538.876212] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4538.891409] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4538.915600] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4539.841496] Lustre: server umount lustre-OST0000 complete [ 4557.791609] Lustre: 139092:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781232615/real 1781232615] req@0000000042947943 x1867757660438528/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1781232631 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4557.842304] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4558.134878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4558.496724] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4558.515789] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4563.617635] Lustre: 139094:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781232621/real 1781232621] req@00000000b735f2a1 x1867757660439808/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1781232637 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4564.417207] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 4565.463570] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4568.207324] Lustre: Failing over lustre-MDT0000 [ 4568.231893] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4568.240568] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4568.256391] Lustre: Skipped 2 previous similar messages [ 4568.266909] Lustre: Skipped 2 previous similar messages [ 4568.468739] Lustre: server umount lustre-MDT0000 complete [ 4569.311600] Lustre: 139094:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781232626/real 1781232626] req@0000000019f0ccb2 x1867757660440832/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1781232642 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4572.319298] Lustre: 139093:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781232630/real 1781232630] req@0000000073fc79b9 x1867757660441728/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1781232646 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4575.688860] LustreError: 140878:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4575.711220] LustreError: 140878:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 4577.765815] LustreError: 140877:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4577.798039] LustreError: 140877:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4580.812831] LustreError: 140878:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4582.893278] LustreError: 140883:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4582.914909] LustreError: 140883:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4586.992584] LustreError: 140877:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4587.012291] LustreError: 140877:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4587.997883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4588.166688] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4588.412706] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 4588.459176] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4588.523545] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4592.078622] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4593.649465] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4593.691626] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4593.700777] Lustre: Skipped 1 previous similar message [ 4593.749585] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4593.957282] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 4603.872444] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4603.873189] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4603.884491] LustreError: Skipped 1 previous similar message [ 4603.895551] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4603.915569] Lustre: Skipped 1 previous similar message [ 4609.000304] LustreError: 144927:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4609.019359] LustreError: 144927:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 4609.123532] Lustre: server umount lustre-OST0000 complete [ 4614.114728] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4614.142521] Lustre: Skipped 1 previous similar message [ 4614.144572] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4614.161433] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4614.173121] Lustre: Skipped 1 previous similar message [ 4619.233339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4619.238782] Lustre: Skipped 1 previous similar message [ 4619.784901] Lustre: server umount lustre-MDT0000 complete [ 4625.679851] LustreError: 140863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781232699 with bad export cookie 13838409240393795434 [ 4625.684442] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4625.690395] LustreError: 140863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4626.392455] Lustre: server umount lustre-MDT0001 complete [ 4640.471240] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 4643.560668] Key type lgssc unregistered [ 4643.990870] LNet: 146101:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4644.002127] LNetError: 146101:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4644.025300] LNet: Removed LNI 192.168.202.154@tcp [ 4644.923408] Key type .llcrypt unregistered [ 4644.925627] Key type ._llcrypt unregistered [ 4681.526532] Key type ._llcrypt registered [ 4681.531951] Key type .llcrypt registered [ 4681.663233] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 4682.771874] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4682.880129] alg: No test for adler32 (adler32-zlib) [ 4683.971888] Lustre: Lustre: Build Version: 2.17.53_64_gab2e0dd [ 4684.215090] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 4685.887175] Key type lgssc registered [ 4687.191394] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4695.215498] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4695.243989] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4696.717136] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4700.955872] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 4706.274580] LustreError: 147608:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4706.298272] LustreError: 147608:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4708.839332] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4711.402420] LustreError: 147607:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4712.658496] LustreError: 147636:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4712.669308] LustreError: 147636:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4712.678284] LustreError: 147636:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 16, retries 0, failed: rc = -5 [ 4712.982357] Lustre: server umount lustre-MDT0000 complete [ 4729.346063] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [ 4741.065047] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 4779.979492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4780.446217] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4780.496521] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4780.636278] Lustre: lustre-MDT0000: new disk, initializing [ 4780.718442] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4780.755294] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4785.108683] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 4795.926987] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4796.084281] Lustre: 150794:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4796.169254] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4796.174257] Lustre: Skipped 1 previous similar message [ 4796.280086] Lustre: lustre-MDT0001: new disk, initializing [ 4796.401967] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4796.452818] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4796.463757] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4800.881444] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 4808.338950] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4812.191269] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4818.471556] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4818.783178] Lustre: lustre-OST0000: new disk, initializing [ 4818.788621] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4818.799684] Lustre: 152536:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4818.901669] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4825.019960] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 4827.740056] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4827.749544] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4827.895518] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4832.712317] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4837.529482] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4838.969832] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 22:55:11 (1781232911) [ 5088.239715] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5088.248580] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5088.267799] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5089.249365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5089.250797] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5091.924431] Lustre: server umount lustre-MDT0000 complete [ 5094.369642] LustreError: 150806:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5094.396036] LustreError: 150806:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5095.568896] LustreError: 151581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781233169 with bad export cookie 10604109203294975043 [ 5095.571175] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5095.579378] LustreError: 151581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5099.489738] LustreError: 150803:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5099.492508] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5099.508449] Lustre: Skipped 1 previous similar message [ 5099.511471] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5099.514427] Lustre: Skipped 2 previous similar messages [ 5102.040213] Lustre: server umount lustre-MDT0001 complete [ 5112.121329] Lustre: server umount lustre-OST0000 complete [ 5121.532879] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [ 5133.201356] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 5175.156178] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5175.542829] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5175.581172] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5175.728326] Lustre: lustre-MDT0000: new disk, initializing [ 5175.901356] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5175.943487] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5180.321572] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 5192.495843] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5192.713791] Lustre: 157051:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5192.844550] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5192.852500] Lustre: Skipped 1 previous similar message [ 5192.929862] Lustre: lustre-MDT0001: new disk, initializing [ 5193.042897] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5193.076789] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5193.103671] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5197.984891] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 5206.156746] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5210.649152] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5217.221877] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5217.545584] Lustre: lustre-OST0000: new disk, initializing [ 5217.549875] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5217.560248] Lustre: 158803:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5217.665825] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5226.625534] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 5227.120556] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5227.136591] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5227.307424] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5235.996074] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5241.491414] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5241.727908] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5244.836338] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5245.163727] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5247.972585] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5247.976961] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5247.977668] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5247.977674] Lustre: Skipped 1 previous similar message [ 5248.002117] Lustre: Skipped 1 previous similar message [ 5248.026869] LustreError: Skipped 1 previous similar message [ 5253.092521] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5253.120691] Lustre: Skipped 1 previous similar message [ 5261.279239] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5261.483503] Lustre: server umount lustre-OST0000 complete [ 5268.451785] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5268.476638] Lustre: Skipped 1 previous similar message [ 5268.476906] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5268.496627] Lustre: Skipped 4 previous similar messages [ 5272.820404] Lustre: server umount lustre-MDT0000 complete [ 5273.571208] LustreError: 157056:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5273.580731] LustreError: 157056:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5276.627787] LustreError: 157042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781233350 with bad export cookie 10604109203295133873 [ 5276.632946] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5276.641567] LustreError: 157042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5276.976467] Lustre: server umount lustre-MDT0001 complete [ 5285.058211] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 23:02:37 (1781233357) [ 5304.115703] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [ 5313.642273] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 5356.408705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5356.582961] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5356.611745] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5356.721336] Lustre: lustre-MDT0000: new disk, initializing [ 5356.801585] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5356.812398] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5363.900455] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 5376.749158] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5376.910453] Lustre: 163382:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5377.252292] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5381.680441] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 5389.665391] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5393.735400] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5402.134531] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5402.576439] Lustre: lustre-OST0000: new disk, initializing [ 5402.587535] Lustre: Skipped 1 previous similar message [ 5402.590937] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5402.593048] Lustre: Skipped 2 previous similar messages [ 5402.600390] Lustre: 165132:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5410.290129] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 5412.499176] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5412.515419] Lustre: Skipped 1 previous similar message [ 5412.521724] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5412.608478] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5419.437237] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5428.196462] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5428.207227] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5428.213075] LustreError: Skipped 1 previous similar message [ 5428.231180] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5428.244058] Lustre: Skipped 1 previous similar message [ 5428.265067] Lustre: Skipped 1 previous similar message [ 5433.802241] Lustre: server umount lustre-OST0000 complete [ 5452.255296] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5452.536731] Lustre: server umount lustre-MDT0000 complete [ 5453.798458] LustreError: 163393:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5453.829765] LustreError: 163393:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5457.079453] LustreError: 163373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781233531 with bad export cookie 10604109203295134748 [ 5457.094370] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5457.099800] LustreError: 163373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5457.455736] Lustre: server umount lustre-MDT0001 complete [ 5467.972919] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 23:05:40 (1781233540) [ 5487.516825] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [ 5497.033495] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 5536.166463] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5536.490655] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5536.527347] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5536.585600] Lustre: lustre-MDT0000: new disk, initializing [ 5536.660788] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5536.669486] Lustre: Skipped 2 previous similar messages [ 5536.682792] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5540.616632] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 5554.437687] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5554.625623] Lustre: 169312:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5554.871230] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5560.380698] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 5567.930369] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5572.009322] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5582.301755] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5582.785889] Lustre: 171062:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5590.078422] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5590.190181] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5590.443410] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 5597.960949] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5605.857332] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5605.861577] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5605.873754] Lustre: Skipped 2 previous similar messages [ 5605.879839] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5605.884527] Lustre: Skipped 8 previous similar messages [ 5611.796716] Lustre: server umount lustre-OST0000 complete [ 5616.610485] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5616.620947] LustreError: Skipped 1 previous similar message [ 5622.789043] Lustre: server umount lustre-MDT0000 complete [ 5626.347966] LustreError: 169317:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5626.386125] LustreError: 169317:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5626.711412] LustreError: 169304:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781233700 with bad export cookie 10604109203295135840 [ 5626.728520] LustreError: 169304:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5626.742805] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5627.624881] Lustre: server umount lustre-MDT0001 complete [ 5639.059498] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 5641.934947] Key type lgssc unregistered [ 5642.308132] LNet: 172788:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5642.318348] LNetError: 172788:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5642.329038] LNet: Removed LNI 192.168.202.154@tcp [ 5643.070413] Key type .llcrypt unregistered [ 5643.073171] Key type ._llcrypt unregistered [ 5669.126694] Key type ._llcrypt registered [ 5669.128245] Key type .llcrypt registered [ 5669.221252] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 5670.886332] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5671.045379] alg: No test for adler32 (adler32-zlib) [ 5672.232723] Lustre: Lustre: Build Version: 2.17.53_64_gab2e0dd [ 5672.497817] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 5674.183190] Key type lgssc registered [ 5675.758193] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5685.716344] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5685.743810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5687.595672] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5692.523753] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 5696.999588] LustreError: 174192:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5697.052996] LustreError: 174192:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5702.113447] LustreError: 174191:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5703.108028] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5703.632212] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5708.361991] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 5716.269230] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5722.352475] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5730.406282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5730.630466] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5732.733269] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5738.050319] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 5744.189866] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5768.673114] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5768.685589] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5768.692706] LustreError: Skipped 1 previous similar message [ 5768.699050] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5768.714986] Lustre: Skipped 1 previous similar message [ 5768.733530] Lustre: Skipped 1 previous similar message [ 5772.415875] Lustre: server umount lustre-OST0000 complete [ 5778.921615] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5778.932462] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5778.945541] Lustre: Skipped 1 previous similar message [ 5778.957122] Lustre: Skipped 1 previous similar message [ 5782.796551] Lustre: server umount lustre-MDT0000 complete [ 5784.032084] LustreError: 174191:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5784.044210] LustreError: 174191:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5786.758688] LustreError: 174170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781233860 with bad export cookie 3575127787296210901 [ 5786.768669] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5786.772506] LustreError: 174170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5787.164202] Lustre: server umount lustre-MDT0001 complete [ 5800.224381] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 5803.569078] Key type lgssc unregistered [ 5803.853524] LNet: 177392:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5803.860374] LNetError: 177392:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5803.898859] LNet: Removed LNI 192.168.202.154@tcp [ 5804.768983] Key type .llcrypt unregistered [ 5804.771859] Key type ._llcrypt unregistered [ 5831.832305] Key type ._llcrypt registered [ 5831.833563] Key type .llcrypt registered [ 5831.955433] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 5833.174673] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5833.382497] alg: No test for adler32 (adler32-zlib) [ 5834.447943] Lustre: Lustre: Build Version: 2.17.53_64_gab2e0dd [ 5834.832119] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 5836.608637] Key type lgssc registered [ 5838.273661] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5847.514157] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5847.538647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5849.104981] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5854.133269] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 5859.301369] LustreError: 178794:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5859.317431] LustreError: 178794:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5864.418064] LustreError: 178795:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5865.415366] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5865.865043] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5871.370377] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 5878.057234] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5883.285904] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5891.896942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5892.402283] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5894.464782] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5899.766886] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 5906.110334] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5918.716878] Lustre: lustre-OST0000: Not available for connect from 192.168.202.54@tcp (stopping) [ 5920.230125] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5920.242075] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5920.271103] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5921.268339] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5921.278276] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5921.285291] Lustre: Skipped 1 previous similar message [ 5921.298615] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5923.794288] Lustre: lustre-OST0000: Not available for connect from 192.168.202.54@tcp (stopping) [ 5928.914858] Lustre: lustre-OST0000: Not available for connect from 192.168.202.54@tcp (stopping) [ 5928.925785] Lustre: Skipped 2 previous similar messages [ 5930.975725] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5931.161412] Lustre: server umount lustre-OST0000 complete [ 5938.502969] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff95d0077a1800.ost_server_uuid 50 [ 5940.493685] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5957.091514] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5957.115766] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5957.123987] Lustre: Skipped 1 previous similar message [ 5962.177240] Lustre: server umount lustre-MDT0000 complete [ 5962.215471] LustreError: 178794:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5962.241297] LustreError: 178794:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5966.425150] LustreError: 178774:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781234040 with bad export cookie 15658850271426853464 [ 5966.444158] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5966.960185] Lustre: server umount lustre-MDT0001 complete [ 5976.490470] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 23:14:08 (1781234048) [ 5984.312098] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5985.043381] LustreError: 182276:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5985.135283] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5990.378791] LustreError: 182277:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5991.014758] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 6000.609890] LustreError: 182276:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6000.625674] LustreError: 182276:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6002.399928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6003.061503] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6007.891759] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 6014.056675] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6018.234685] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6026.463449] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6026.862737] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6030.980142] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6033.975272] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 6041.055136] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6048.518285] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6048.842473] Lustre: lustre-OST0001: new disk, initializing [ 6048.848562] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6048.852704] Lustre: 184781:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6048.926974] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6054.790131] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 6057.095705] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6057.100394] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6057.214081] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6063.459308] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6067.685611] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6067.701934] Lustre: Skipped 1 previous similar message [ 6067.712372] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6072.288139] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6072.291549] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6074.028761] Lustre: server umount lustre-OST0000 complete [ 6074.839824] LustreError: 183873:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6080.620514] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6080.857754] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6085.869831] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6086.099948] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6103.521161] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6103.529224] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6103.539657] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6103.541244] Lustre: Skipped 1 previous similar message [ 6103.547701] Lustre: Skipped 3 previous similar messages [ 6105.253961] Lustre: server umount lustre-OST0001 complete [ 6110.691239] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6110.696383] LustreError: Skipped 1 previous similar message [ 6116.075616] Lustre: server umount lustre-MDT0000 complete [ 6118.882941] LustreError: 182276:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6118.900521] LustreError: 182276:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 17 previous similar messages [ 6120.288259] LustreError: 182257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781234194 with bad export cookie 15658850271426854535 [ 6120.291475] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6120.308649] LustreError: 182257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 6120.713527] Lustre: server umount lustre-MDT0001 complete [ 6142.106737] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 23:16:54 (1781234214) [ 6148.851937] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6149.137509] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6149.171946] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6149.322937] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6153.143732] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 6163.040144] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6163.234449] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6163.311144] Lustre: 188076:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6169.131456] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 6176.237304] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6181.913350] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6189.620902] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6189.760880] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6197.454880] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 6199.073191] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 6203.086257] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6210.622946] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6210.703959] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6217.681899] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 6219.120728] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6223.761538] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6229.499496] Lustre: lustre-OST0000: Not available for connect from 192.168.202.54@tcp (stopping) [ 6229.502119] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6229.506977] Lustre: Skipped 4 previous similar messages [ 6229.533815] Lustre: Skipped 3 previous similar messages [ 6232.834190] Lustre: server umount lustre-OST0000 complete [ 6234.573303] LustreError: 189702:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6255.075713] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6255.090165] LustreError: Skipped 1 previous similar message [ 6256.881720] Lustre: server umount lustre-OST0001 complete [ 6265.316974] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6265.323380] Lustre: Skipped 3 previous similar messages [ 6267.279078] Lustre: server umount lustre-MDT0000 complete [ 6271.165962] LustreError: 188070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781234345 with bad export cookie 15658850271426855942 [ 6271.173874] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6271.517813] Lustre: server umount lustre-MDT0001 complete [ 6296.574688] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 23:19:28 (1781234368) [ 6313.862912] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [ 6322.797568] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 6366.348881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6366.631433] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6366.658272] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6366.729696] Lustre: lustre-MDT0000: new disk, initializing [ 6366.801426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6366.805264] Lustre: Skipped 3 previous similar messages [ 6366.832020] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6371.791374] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 6384.934988] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6385.130949] Lustre: 195687:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6385.233725] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6385.239071] Lustre: Skipped 1 previous similar message [ 6385.297106] Lustre: lustre-MDT0001: new disk, initializing [ 6385.494213] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6385.502238] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6391.204528] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 6399.567875] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6403.809516] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6411.438404] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6411.834535] Lustre: lustre-OST0000: new disk, initializing [ 6411.839404] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6411.843228] Lustre: 197430:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6418.557226] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 6421.071625] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6421.078333] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6421.184707] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6426.425575] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6431.785673] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6432.032670] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6437.855548] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6438.203975] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6441.956544] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6441.959440] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6441.998521] LustreError: Skipped 1 previous similar message [ 6441.999384] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6442.034870] Lustre: Skipped 8 previous similar messages [ 6446.908609] Lustre: server umount lustre-OST0000 complete [ 6457.363419] LustreError: 195694:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6457.378463] LustreError: 195694:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 15 previous similar messages [ 6457.392632] Lustre: server umount lustre-MDT0000 complete [ 6462.199310] LustreError: 195679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781234536 with bad export cookie 15658850271426857727 [ 6462.205389] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6462.217968] LustreError: 195679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 6463.042134] Lustre: server umount lustre-MDT0001 complete [ 6473.634762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6480.077578] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 6490.205919] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6495.650947] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 6502.388537] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6506.224466] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6514.096144] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6514.496514] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6514.506108] Lustre: Skipped 4 previous similar messages [ 6521.789547] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 6527.895308] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6533.197704] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6533.457976] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6538.521332] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6538.690989] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6541.280717] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6541.297675] LustreError: Skipped 1 previous similar message [ 6555.104394] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6555.231799] Lustre: server umount lustre-OST0000 complete [ 6561.995796] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6568.000946] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6598.653202] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6600.057979] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6600.095996] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6605.234391] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6605.257878] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6606.267889] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 6612.981106] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6616.062460] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec [ 6620.647819] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6620.661783] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6620.668392] Lustre: Skipped 6 previous similar messages [ 6620.686917] LustreError: Skipped 1 previous similar message [ 6626.614245] Lustre: server umount lustre-OST0000 complete [ 6636.746866] Lustre: server umount lustre-MDT0000 complete [ 6640.804688] LustreError: 199482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781234714 with bad export cookie 15658850271426858609 [ 6640.815507] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6641.082885] Lustre: server umount lustre-MDT0001 complete [ 6651.401113] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 23:25:23 (1781234723) [ 6658.192411] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6662.750190] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 6671.469357] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6677.355276] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 6684.094382] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6689.849868] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6698.425481] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6706.116704] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 6713.495455] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6718.586515] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6718.860984] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6724.367873] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6724.745778] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6732.526432] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6732.845369] Lustre: lustre-OST0001: new disk, initializing [ 6732.848908] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6732.854943] Lustre: 207950:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6740.155232] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 6742.152494] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6742.170679] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6742.262457] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6748.310559] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6753.522421] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6753.795501] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6758.370851] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6758.625516] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6762.978578] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6762.996803] LustreError: Skipped 1 previous similar message [ 6763.025449] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6763.028348] Lustre: Skipped 16 previous similar messages [ 6766.954835] Lustre: server umount lustre-OST0001 complete [ 6768.123390] LustreError: 206547:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6768.150848] LustreError: 206547:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 6774.578963] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6774.835284] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6780.650917] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6781.027350] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6811.596865] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6811.891953] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6811.900752] Lustre: Skipped 5 previous similar messages [ 6817.277251] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6817.298787] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6817.302235] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6818.230395] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 6823.970349] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6825.486788] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6833.418174] Lustre: server umount lustre-OST0001 complete [ 6865.375423] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6865.741339] Lustre: server umount lustre-OST0000 complete [ 6878.762878] Lustre: server umount lustre-MDT0000 complete [ 6883.549741] LustreError: 204931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781234957 with bad export cookie 15658850271426859575 [ 6883.565023] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6883.570719] LustreError: 204931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 6890.193583] Lustre: server umount lustre-MDT0001 complete [ 6916.499838] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 23:29:48 (1781234988) [ 6923.824308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6924.069897] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6924.094944] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6930.701909] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 6942.319991] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6942.504918] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6942.546774] Lustre: 212901:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6948.906534] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 6955.764546] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6960.094365] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6966.764565] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6967.043538] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6974.325977] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 6981.227497] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6992.303231] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6992.406917] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7000.393984] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 7008.038658] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7010.175330] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7015.319159] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7015.643971] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7020.989424] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7021.267487] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7025.682055] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff95d0077a6000.ost_server_uuid 50 [ 7027.380915] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff95d0077a6000.ost_server_uuid in IDLE state after 0 sec [ 7028.400681] Lustre: Permanently deactivating lustre-OST0001 [ 7028.418739] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7032.282668] Lustre: Permanently reactivating lustre-OST0001 [ 7037.920202] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7037.930200] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7037.932395] LustreError: Skipped 4 previous similar messages [ 7037.969282] Lustre: Skipped 11 previous similar messages [ 7039.982441] Lustre: server umount lustre-OST0001 complete [ 7050.773641] Lustre: server umount lustre-OST0000 complete [ 7062.244606] Lustre: server umount lustre-MDT0000 complete [ 7067.564733] LustreError: 212895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781235141 with bad export cookie 15658850271426860863 [ 7067.570090] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7067.579725] LustreError: 212895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7067.873962] Lustre: server umount lustre-MDT0001 complete [ 7093.041501] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 23:32:45 (1781235165) [ 7101.094075] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7101.347822] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7101.378429] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7101.387616] Lustre: Skipped 5 previous similar messages [ 7105.395956] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 7114.821153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7114.954582] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7114.998621] Lustre: 219018:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7120.285303] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 7126.605991] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7130.785178] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7138.173600] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7138.376475] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7138.394844] Lustre: Permanently deactivating lustre-OST0000 [ 7138.407799] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7145.378582] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 7147.573865] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7151.277709] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7159.455354] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7159.492937] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7165.227194] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7168.316165] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 7175.243173] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7177.547340] Lustre: Permanently reactivating lustre-OST0000 [ 7177.565271] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7177.569905] Lustre: Skipped 2 previous similar messages [ 7183.521158] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 7183.547077] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 36 seconds. I think it's dead, and I am evicting it. exp 0000000085009ed3, cur 1781235257 deadline 1781235251 last 1781235221 [ 7183.654229] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7207.391180] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7207.604310] Lustre: server umount lustre-OST0001 complete [ 7226.336880] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7226.569241] Lustre: server umount lustre-OST0000 complete [ 7237.588970] Lustre: server umount lustre-MDT0000 complete [ 7243.908132] LustreError: 219741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781235317 with bad export cookie 15658850271426862543 [ 7243.922361] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7243.923683] LustreError: 219741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7244.632097] Lustre: server umount lustre-MDT0001 complete [ 7256.644571] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 7259.908056] Key type lgssc unregistered [ 7260.333463] LNet: 223429:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7260.348703] LNetError: 223429:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7260.368640] LNet: Removed LNI 192.168.202.154@tcp [ 7261.371858] Key type .llcrypt unregistered [ 7261.377287] Key type ._llcrypt unregistered [ 7287.004795] Key type ._llcrypt registered [ 7287.007540] Key type .llcrypt registered [ 7287.119683] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 7288.713820] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7288.783176] alg: No test for adler32 (adler32-zlib) [ 7289.983328] Lustre: Lustre: Build Version: 2.17.53_64_gab2e0dd [ 7290.393821] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 7292.119188] Key type lgssc registered [ 7293.480954] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7304.784181] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7304.811210] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7306.400932] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7311.571768] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 7316.960782] LustreError: 224881:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7316.999265] LustreError: 224881:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7322.085719] LustreError: 224882:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7322.313144] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7322.460674] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7322.491181] Lustre: 224871:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7322.502924] Lustre: Permanently deactivating lustre-MDT0001 [ 7322.519662] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7322.869226] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7328.297765] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 7329.824714] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7336.139776] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7341.553347] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7349.936518] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7350.321264] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7351.340923] LustreError: 226497:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7351.363102] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7356.402062] LustreError: 226497:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7358.599616] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 7361.510398] LustreError: 226557:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7366.359876] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7367.743256] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7372.788757] LustreError: 226557:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7372.821165] LustreError: 226557:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 7375.403603] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7375.769491] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7380.996422] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7381.006081] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7382.830416] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 7389.264484] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7392.777383] Lustre: Permanently reactivating lustre-MDT0001 [ 7392.792387] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7392.801936] Lustre: Skipped 2 previous similar messages [ 7396.618455] Lustre: lustre-MDT0001: Client a536269b-bece-41c2-8896-3bfd9ce386ee (at 192.168.202.54@tcp) reconnecting [ 7401.039217] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7403.147129] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7403.171267] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7403.191657] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7403.210395] LustreError: 228113:0:(llog_osd.c:2259:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 7403.224322] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7403.225059] LustreError: 228113:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 7403.225221] LustreError: 228113:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.154@tcp: cfg command failed: rc = -108 [ 7403.225243] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 7403.225243] [ 7403.226358] LustreError: 224873:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 7403.526689] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7408.890785] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7409.035678] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7413.886464] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7414.252390] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7417.139729] Lustre: Permanently deactivating lustre-MDT0001 [ 7417.155743] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7417.160280] Lustre: Skipped 2 previous similar messages [ 7426.213315] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7438.311196] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7438.317316] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7438.332750] Lustre: Skipped 3 previous similar messages [ 7438.342508] Lustre: Skipped 3 previous similar messages [ 7443.437806] LustreError: 225845:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7443.879155] Lustre: server umount lustre-MDT0000 complete [ 7449.811976] LustreError: 228786:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781235523 with bad export cookie 8139896099878368093 [ 7449.814444] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7449.850238] LustreError: 228786:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7450.750835] Lustre: server umount lustre-MDT0001 complete [ 7456.207146] Lustre: server umount lustre-OST0000 complete [ 7471.274739] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 23:39:03 (1781235543) [ 7484.025818] Lustre: server umount lustre-OST0001 complete [ 7491.506911] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [ 7500.985855] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 7542.726100] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7543.007189] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7543.024704] Lustre: Skipped 2 previous similar messages [ 7543.053598] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7543.147817] Lustre: lustre-MDT0000: new disk, initializing [ 7543.222657] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7543.241963] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7547.806902] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 7558.711789] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7558.847348] Lustre: 232582:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7558.929144] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7558.934286] Lustre: Skipped 1 previous similar message [ 7559.020332] Lustre: lustre-MDT0001: new disk, initializing [ 7559.120910] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7559.160165] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7559.175738] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7563.967888] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 7571.447777] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7576.544961] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7584.686373] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7585.015580] Lustre: lustre-OST0000: new disk, initializing [ 7585.020887] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7585.035689] Lustre: 234323:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7585.134572] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7591.889030] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 7592.487076] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7592.492461] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7592.688370] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7599.664751] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7602.078790] LustreError: 232590:0:(mdt_handler.c:3266:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7611.545762] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7611.827599] Lustre: lustre-OST0001: new disk, initializing [ 7611.837851] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7611.854769] Lustre: 235324:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7612.079501] LustreError: 232590:0:(mdt_handler.c:3266:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7612.141042] LustreError: 235139:0:(mdt_handler.c:3266:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7619.168779] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 7621.840063] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7621.845944] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7621.917810] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7622.159723] LustreError: 235139:0:(mdt_handler.c:3266:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7627.413659] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7629.332357] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7631.924371] Lustre: lustre-OST0001: Not available for connect from 192.168.202.54@tcp (stopping) [ 7632.365321] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7632.374224] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7636.939863] Lustre: lustre-OST0001: Not available for connect from 192.168.202.54@tcp (stopping) [ 7636.953461] Lustre: Skipped 2 previous similar messages [ 7637.873762] Lustre: server umount lustre-OST0001 complete [ 7642.085368] LustreError: 234309:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7642.109275] LustreError: 234309:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 7657.969968] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7657.981276] Lustre: Skipped 2 previous similar messages [ 7657.981841] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7658.007719] Lustre: Skipped 3 previous similar messages [ 7662.441904] Lustre: server umount lustre-OST0000 complete [ 7668.195328] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7668.223309] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7673.445616] Lustre: server umount lustre-MDT0000 complete [ 7677.971874] LustreError: 232574:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781235751 with bad export cookie 8139896099878370907 [ 7677.982851] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7677.984787] LustreError: 232574:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7684.440441] Lustre: server umount lustre-MDT0001 complete [ 7698.164129] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 7701.035299] Key type lgssc unregistered [ 7701.380226] LNet: 237229:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7701.396288] LNetError: 237229:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7701.408687] LNet: Removed LNI 192.168.202.154@tcp [ 7702.371782] Key type .llcrypt unregistered [ 7702.374867] Key type ._llcrypt unregistered [ 7748.741784] Key type ._llcrypt registered [ 7748.745674] Key type .llcrypt registered [ 7748.892120] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 7750.469028] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7750.552648] alg: No test for adler32 (adler32-zlib) [ 7751.737289] Lustre: Lustre: Build Version: 2.17.53_64_gab2e0dd [ 7751.943090] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 7753.619307] Key type lgssc registered [ 7754.548142] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7763.245384] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7763.260196] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7764.631967] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7764.689419] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7764.903131] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7770.573810] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 7780.697276] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7780.859898] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7780.894377] Lustre: 239419:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7781.260335] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7786.874747] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 7793.129798] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7797.706163] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7804.734718] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7804.974297] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7805.223504] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7810.849738] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 7812.259819] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7816.586230] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7868.897801] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7868.925146] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7873.517800] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7873.524234] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7873.536144] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7875.446729] Lustre: server umount lustre-OST0000 complete [ 7883.241489] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7902.606427] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7903.469529] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7911.705223] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 7918.409139] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7924.966806] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7924.983959] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7924.987668] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7924.998799] Lustre: Skipped 1 previous similar message [ 7961.056597] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7961.073194] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7961.101447] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7961.114463] Lustre: Skipped 1 previous similar message [ 7961.130879] Lustre: Skipped 3 previous similar messages [ 7965.410897] Lustre: server umount lustre-OST0000 complete [ 7970.784549] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7970.791884] LustreError: Skipped 1 previous similar message [ 7970.796235] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7970.805870] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7976.417543] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7976.420364] Lustre: Skipped 2 previous similar messages [ 7984.095171] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7984.579949] Lustre: server umount lustre-MDT0000 complete [ 7988.920493] LustreError: 241055:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781236062 with bad export cookie 11665212323927663089 [ 7988.924203] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7988.928447] LustreError: 241055:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7989.317341] Lustre: server umount lustre-MDT0001 complete [ 8001.529482] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 8004.257622] Key type lgssc unregistered [ 8004.486874] LNet: 244379:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8004.490885] LNetError: 244379:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8004.502151] LNet: Removed LNI 192.168.202.154@tcp [ 8005.292521] Key type .llcrypt unregistered [ 8005.294229] Key type ._llcrypt unregistered [ 8035.362590] Key type ._llcrypt registered [ 8035.369656] Key type .llcrypt registered [ 8035.516995] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 8037.094926] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8037.262263] alg: No test for adler32 (adler32-zlib) [ 8038.468624] Lustre: Lustre: Build Version: 2.17.53_64_gab2e0dd [ 8038.756146] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 8040.527197] Key type lgssc registered [ 8042.071983] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8051.605075] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8051.635443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8053.338935] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8058.633379] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 8063.460425] LustreError: 245780:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8063.498111] LustreError: 245780:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8068.577107] LustreError: 245779:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8068.614990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8069.080497] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8074.054130] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 8080.570901] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8085.323562] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8093.318955] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8093.818803] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8098.936666] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8100.026096] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 8107.471407] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8140.257432] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8140.260037] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8140.272400] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8140.292602] Lustre: Skipped 1 previous similar message [ 8140.301273] Lustre: Skipped 1 previous similar message [ 8144.340893] Lustre: server umount lustre-OST0000 complete [ 8150.505312] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8150.507079] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8150.524750] Lustre: Skipped 1 previous similar message [ 8150.549100] Lustre: Skipped 1 previous similar message [ 8155.618699] LustreError: 246988:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8155.644733] LustreError: 246988:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8155.958226] Lustre: server umount lustre-MDT0000 complete [ 8160.743930] LustreError: 245774:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8160.767340] LustreError: 245774:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8160.772167] LustreError: 245761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781236234 with bad export cookie 14359962413835958799 [ 8160.782656] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8161.407718] Lustre: server umount lustre-MDT0001 complete [ 8176.744603] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 8179.911421] Key type lgssc unregistered [ 8180.341075] LNet: 249732:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8180.350440] LNetError: 249732:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8180.372335] LNet: Removed LNI 192.168.202.154@tcp [ 8181.382773] Key type .llcrypt unregistered [ 8181.384821] Key type ._llcrypt unregistered [ 8201.483517] Key type ._llcrypt registered [ 8201.485496] Key type .llcrypt registered [ 8201.588924] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 8202.569247] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8202.605222] alg: No test for adler32 (adler32-zlib) [ 8203.707798] Lustre: Lustre: Build Version: 2.17.53_64_gab2e0dd [ 8203.904910] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 8205.559233] Key type lgssc registered [ 8207.638473] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8219.747989] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8219.780357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8221.495207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8227.078788] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 8231.907349] LustreError: 250937:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8231.922038] LustreError: 250937:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8237.034527] LustreError: 250938:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8237.836636] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8238.262828] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8243.988869] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 8251.290906] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8256.166393] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8264.989265] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8265.380863] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8266.481928] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8275.903813] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 8284.050970] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8299.502307] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8299.509977] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8299.544271] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8299.555931] Lustre: Skipped 1 previous similar message [ 8299.578052] Lustre: Skipped 1 previous similar message [ 8304.619133] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8304.636524] Lustre: Skipped 1 previous similar message [ 8305.180174] Lustre: server umount lustre-OST0000 complete [ 8314.849704] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8314.856550] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8314.882990] Lustre: Skipped 1 previous similar message [ 8317.413686] Lustre: server umount lustre-MDT0000 complete [ 8319.968946] LustreError: 252591:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8319.999502] LustreError: 252591:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8322.603637] LustreError: 250918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781236396 with bad export cookie 3601851410176431767 [ 8322.604853] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8322.623494] LustreError: 250918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8323.472914] Lustre: server umount lustre-MDT0001 complete [ 8337.964673] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 8341.603673] Key type lgssc unregistered [ 8341.854571] LNet: 254262:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8341.869682] LNetError: 254262:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8341.888698] LNet: Removed LNI 192.168.202.154@tcp [ 8342.782147] Key type .llcrypt unregistered [ 8342.783707] Key type ._llcrypt unregistered [ 8369.632637] Key type ._llcrypt registered [ 8369.635127] Key type .llcrypt registered [ 8369.744712] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 8371.032969] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8371.235156] alg: No test for adler32 (adler32-zlib) [ 8372.420030] Lustre: Lustre: Build Version: 2.17.53_64_gab2e0dd [ 8372.698560] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 8374.399237] Key type lgssc registered [ 8376.032567] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8386.200200] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8386.227312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8388.231062] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8393.859875] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 8398.818894] LustreError: 255662:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8398.844016] LustreError: 255662:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8403.938101] LustreError: 255663:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8405.867920] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8406.226307] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8411.817513] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 8418.891463] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8423.567529] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8432.110668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8432.450456] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8433.475685] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8440.202469] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 8447.347603] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8479.722848] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8479.732342] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8479.743103] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8482.793915] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8482.795203] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8482.800237] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8484.315827] Lustre: server umount lustre-OST0000 complete [ 8493.024993] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8493.045168] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8493.051551] Lustre: Skipped 2 previous similar messages [ 8495.506354] Lustre: server umount lustre-MDT0000 complete [ 8498.144287] LustreError: 255659:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8498.165593] LustreError: 255659:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8499.866184] LustreError: 255644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781236573 with bad export cookie 3355812406063453230 [ 8499.867451] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8499.881562] LustreError: 255644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8500.295418] Lustre: server umount lustre-MDT0001 complete [ 8512.104386] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 8515.428607] Key type lgssc unregistered [ 8515.629568] LNet: 259615:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8515.637704] LNetError: 259615:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8515.660109] LNet: Removed LNI 192.168.202.154@tcp [ 8516.852235] Key type .llcrypt unregistered [ 8516.856093] Key type ._llcrypt unregistered [ 8537.582839] Key type ._llcrypt registered [ 8537.585694] Key type .llcrypt registered [ 8537.733810] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 8539.119755] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8539.539624] alg: No test for adler32 (adler32-zlib) [ 8540.851067] Lustre: Lustre: Build Version: 2.17.53_64_gab2e0dd [ 8541.135865] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 8542.815274] Key type lgssc registered [ 8544.492493] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8554.763077] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8554.780467] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8556.855703] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8562.864807] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 8566.244867] LustreError: 260835:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8566.262126] LustreError: 260835:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8571.364025] LustreError: 260837:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8573.700916] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8574.232405] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8579.687301] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 8586.825044] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8591.261464] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8599.372042] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8600.229534] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8601.294052] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8608.370933] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 8615.115329] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8632.289124] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8632.298079] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8632.309782] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8635.377700] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8635.379850] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8635.399971] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8638.001758] Lustre: server umount lustre-OST0000 complete [ 8645.604951] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8645.611912] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8645.623873] Lustre: Skipped 1 previous similar message [ 8645.638986] Lustre: Skipped 2 previous similar messages [ 8650.721618] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8650.738794] Lustre: Skipped 1 previous similar message [ 8655.842198] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8655.849597] Lustre: Skipped 1 previous similar message [ 8656.352506] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8656.689446] Lustre: server umount lustre-MDT0000 complete [ 8660.543414] LustreError: 260803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781236734 with bad export cookie 975666393027017477 [ 8660.552291] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8661.023260] Lustre: server umount lustre-MDT0001 complete [ 8671.363818] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 8674.026135] Key type lgssc unregistered [ 8674.391311] LNet: 264184:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8674.410622] LNetError: 264184:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8674.424728] LNet: Removed LNI 192.168.202.154@tcp [ 8675.133958] Key type .llcrypt unregistered [ 8675.136446] Key type ._llcrypt unregistered [ 8686.562718] Key type ._llcrypt registered [ 8686.568254] Key type .llcrypt registered [ 8686.776752] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8708.454440] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [ 8724.340195] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 8726.175283] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8726.303147] alg: No test for adler32 (adler32-zlib) [ 8727.604427] Lustre: Lustre: Build Version: 2.17.53_64_gab2e0dd [ 8727.870481] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 8729.567181] Key type lgssc registered [ 8730.827380] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8771.227631] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8771.275480] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8772.621983] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8772.682155] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8772.817382] Lustre: lustre-MDT0000: new disk, initializing [ 8772.985167] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8773.016423] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8778.190404] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 8790.623591] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8790.839468] Lustre: 267689:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8790.944189] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8790.949251] Lustre: Skipped 1 previous similar message [ 8791.068425] Lustre: lustre-MDT0001: new disk, initializing [ 8791.186885] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8791.231684] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8791.242229] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8796.180867] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 8804.221172] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8808.852320] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8817.781896] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8818.185753] Lustre: lustre-OST0000: new disk, initializing [ 8818.192264] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8818.198172] Lustre: 269445:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8818.321282] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8825.358299] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 8826.463295] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8826.481203] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8826.635604] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8834.147992] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8839.472176] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8839.809069] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8845.357271] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8845.583268] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8852.451299] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8852.452979] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8852.461098] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8852.482798] Lustre: Skipped 1 previous similar message [ 8852.505278] LustreError: Skipped 1 previous similar message [ 8853.893258] Lustre: server umount lustre-OST0000 complete [ 8858.081459] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8858.090474] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8858.106434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8858.108661] Lustre: Skipped 1 previous similar message [ 8862.690876] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8862.691956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8864.432739] Lustre: server umount lustre-MDT0000 complete [ 8868.435543] LustreError: 268482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781236942 with bad export cookie 5757294573561105504 [ 8868.438259] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8868.446812] LustreError: 268482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8869.083815] Lustre: server umount lustre-MDT0001 complete [ 8879.734758] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 00:02:31 (1781236951) [ 8887.394346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8887.929154] LustreError: 271713:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8887.950833] LustreError: 271713:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8888.031091] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8893.408803] LustreError: 271714:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8893.425246] LustreError: 271714:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8893.548065] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 8897.509749] LustreError: 271713:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8902.630968] LustreError: 271714:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8903.602542] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8904.138489] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8908.562052] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 8914.961941] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8919.096510] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8926.787494] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8927.172126] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8933.874595] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 8941.079638] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8954.851736] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8954.860060] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8954.875271] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8954.877431] Lustre: Skipped 1 previous similar message [ 8957.921108] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8959.224952] Lustre: server umount lustre-OST0000 complete [ 8965.602758] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8965.615279] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8965.643682] Lustre: Skipped 1 previous similar message [ 8965.646713] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8965.649544] Lustre: Skipped 2 previous similar messages [ 8970.067347] Lustre: server umount lustre-MDT0000 complete [ 8973.280186] LustreError: 271714:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8973.291929] LustreError: 271714:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8973.976308] LustreError: 271693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781237047 with bad export cookie 5757294573561106386 [ 8973.986933] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8973.992215] LustreError: 271693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8974.449768] Lustre: server umount lustre-MDT0001 complete [ 8990.332597] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 8994.504413] Key type lgssc unregistered [ 8994.769462] LNet: 274891:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8994.777088] LNetError: 274891:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8994.798947] LNet: Removed LNI 192.168.202.154@tcp [ 8996.396752] Key type .llcrypt unregistered [ 8996.400922] Key type ._llcrypt unregistered [ 9027.388373] Key type ._llcrypt registered [ 9027.392628] Key type .llcrypt registered [ 9027.552594] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [ 9043.852335] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 9045.904448] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9045.971380] alg: No test for adler32 (adler32-zlib) [ 9047.055354] Lustre: Lustre: Build Version: 2.17.53_64_gab2e0dd [ 9047.332397] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 9048.991215] Key type lgssc registered [ 9050.486045] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9100.445967] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9100.461389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9101.776604] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9101.815975] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9101.920628] Lustre: lustre-MDT0000: new disk, initializing [ 9102.064579] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9102.088986] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9106.676892] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 9118.935400] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9119.212688] Lustre: 278685:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9119.328899] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9119.338953] Lustre: Skipped 1 previous similar message [ 9119.508211] Lustre: lustre-MDT0001: new disk, initializing [ 9119.706735] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9119.754905] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9119.766141] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9124.747076] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 9134.015661] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9138.223951] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9147.009393] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9147.673368] Lustre: lustre-OST2710: new disk, initializing [ 9147.686240] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9147.695840] Lustre: 280428:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9147.805774] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9155.633960] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9155.647294] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9155.838263] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9155.935664] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 9165.137156] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 9173.640590] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9174.036728] Lustre: lustre-OST03e8: new disk, initializing [ 9174.047174] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9174.067823] Lustre: 281381:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9174.250422] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9180.430521] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9180.441663] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9180.533370] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9184.024867] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 9191.051219] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 9199.169552] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9199.441510] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9205.053384] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9205.340068] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9212.886875] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9213.329642] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9219.854885] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9220.197888] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9226.727350] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9226.732726] Lustre: Skipped 3 previous similar messages [ 9226.808513] Lustre: server umount lustre-MDT0000 complete [ 9231.192126] LustreError: 281380:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781237305 with bad export cookie 6566148686973797507 [ 9231.195891] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9231.203389] LustreError: 281380:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9231.628743] Lustre: server umount lustre-MDT0001 complete [ 9245.200466] Lustre: server umount lustre-OST2710 complete [ 9247.201540] Lustre: 276483:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781237305/real 1781237305] req@0000000037625701 x1867762495463168/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781237321 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9247.235645] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9250.185577] Lustre: server umount lustre-OST03e8 complete [ 9272.503546] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [ 9285.331176] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 9329.977806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9330.217841] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9330.237610] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9330.292763] Lustre: lustre-MDT0000: new disk, initializing [ 9330.366788] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9330.382338] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9335.147322] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 9347.362764] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9347.543662] Lustre: 286420:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9347.601094] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9347.605271] Lustre: Skipped 1 previous similar message [ 9347.691953] Lustre: lustre-MDT0001: new disk, initializing [ 9347.798381] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9347.849208] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9347.860543] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9353.179885] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 9362.124489] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9366.304672] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9374.090829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9374.418827] Lustre: lustre-OST0000: new disk, initializing [ 9374.422164] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9374.436193] Lustre: 288161:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9374.527275] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9380.908746] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 9382.017986] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9382.028926] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9382.205347] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9389.396216] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9395.028394] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9395.205684] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9400.362595] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9400.618022] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9402.856199] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9402.868510] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9402.883620] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9402.885029] LustreError: Skipped 1 previous similar message [ 9402.899326] Lustre: Skipped 1 previous similar message [ 9407.998617] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9408.008911] Lustre: Skipped 1 previous similar message [ 9413.091743] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9417.184806] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9417.491435] Lustre: server umount lustre-OST0000 complete [ 9423.329915] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9423.347374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9423.354890] Lustre: Skipped 2 previous similar messages [ 9423.373426] Lustre: Skipped 2 previous similar messages [ 9428.451248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9436.128174] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9436.428922] Lustre: server umount lustre-MDT0000 complete [ 9438.688880] LustreError: 286430:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9438.721903] LustreError: 286430:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 9440.707330] LustreError: 286411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781237514 with bad export cookie 6566148686973799187 [ 9440.720101] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9440.722822] LustreError: 286411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9441.202294] Lustre: server umount lustre-MDT0001 complete [ 9451.966802] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 00:12:04 (1781237524) [ 9454.079226] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9456.053452] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 00:12:08 (1781237528) [ 9459.639856] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 9489.794792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9490.249257] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9490.308163] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9490.570171] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9495.462474] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 9505.295608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9505.363822] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9505.396225] Lustre: 291855:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9510.929896] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 9537.791423] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9538.215135] Lustre: Denying initial registration attempt from nid 192.168.202.154@tcp, specified as failover [ 9538.232537] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9538.247184] LustreError: 293278:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -99 [ 9538.266198] LustreError: 293278:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 9538.281299] LustreError: 293278:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9538.491411] Lustre: server umount lustre-OST0000 complete [ 9538.493204] LustreError: 293278:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9548.259419] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9548.272614] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9548.275472] Lustre: Skipped 3 previous similar messages [ 9553.377545] LustreError: 291862:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9553.408223] LustreError: 291862:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9553.473292] Lustre: server umount lustre-MDT0000 complete [ 9557.890056] LustreError: 291848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781237631 with bad export cookie 6566148686973800069 [ 9557.892502] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9557.897543] LustreError: 291848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 9558.310408] Lustre: server umount lustre-MDT0001 complete [ 9570.991433] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 9575.394355] Key type lgssc unregistered [ 9575.829608] LNet: 294380:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9575.838431] LNetError: 294380:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9575.853562] LNet: Removed LNI 192.168.202.154@tcp [ 9577.173211] Key type .llcrypt unregistered [ 9577.177251] Key type ._llcrypt unregistered [ 9589.810693] Key type ._llcrypt registered [ 9589.813775] Key type .llcrypt registered [ 9589.974319] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 9591.423777] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9591.453375] alg: No test for adler32 (adler32-zlib) [ 9592.563113] Lustre: Lustre: Build Version: 2.17.53_64_gab2e0dd [ 9592.869844] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 9594.551451] Key type lgssc registered [ 9596.131857] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9650.429751] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 9662.070851] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9662.096418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9663.620667] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9663.684380] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9663.934721] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9669.211375] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 9679.424663] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9679.587413] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9679.663173] Lustre: 297112:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9680.056120] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9685.616274] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 9692.808466] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9697.139514] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9718.199117] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9718.588192] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9719.073082] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9726.027755] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 9732.545650] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9738.210199] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9738.226735] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9738.260765] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9738.720684] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9738.734155] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9738.769378] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9741.677410] Lustre: server umount lustre-OST0000 complete [ 9746.912851] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9746.919356] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9746.933569] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9748.962363] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9748.965360] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9753.077637] Lustre: server umount lustre-MDT0000 complete [ 9757.452497] LustreError: 297104:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781237831 with bad export cookie 9879553631817330336 [ 9757.457200] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9757.477178] LustreError: 297104:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9758.019443] Lustre: server umount lustre-MDT0001 complete [ 9770.137249] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 9774.728579] Key type lgssc unregistered [ 9775.170731] LNet: 300335:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9775.184976] LNetError: 300335:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9775.218289] LNet: Removed LNI 192.168.202.154@tcp [ 9776.106778] Key type .llcrypt unregistered [ 9776.115557] Key type ._llcrypt unregistered [ 9806.285408] Key type ._llcrypt registered [ 9806.288308] Key type .llcrypt registered [ 9806.445525] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 9808.348339] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9808.370185] alg: No test for adler32 (adler32-zlib) [ 9809.698714] Lustre: Lustre: Build Version: 2.17.53_64_gab2e0dd [ 9810.229770] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 9812.007844] Key type lgssc registered [ 9813.475212] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9823.149975] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9823.167711] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9824.720665] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9829.556707] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 9834.977769] LustreError: 301738:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9834.997063] LustreError: 301738:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9839.338070] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9839.813667] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9844.736732] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 9850.773808] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9856.095511] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9865.928624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9866.337210] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9874.563495] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 9880.887976] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9895.394547] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9895.398427] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9895.418514] Lustre: Skipped 2 previous similar messages [ 9895.451783] Lustre: Skipped 2 previous similar messages [ 9898.996253] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.54@tcp (stopping) [ 9900.512911] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9900.540609] Lustre: Skipped 2 previous similar messages [ 9901.764067] Lustre: server umount lustre-MDT0000 complete [ 9904.084128] LustreError: 301733:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9905.635361] LustreError: 301738:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9906.535569] LustreError: 301720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781237980 with bad export cookie 383247876642589972 [ 9906.546185] LustreError: 301720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9906.546900] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9909.198047] LustreError: 303893:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9909.233950] LustreError: 303893:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9909.258173] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.54@tcp (stopping) [ 9910.754713] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9913.153889] Lustre: server umount lustre-MDT0001 complete [ 9919.747925] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9930.629822] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9931.710781] LustreError: 303345:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9931.732140] LustreError: 303345:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9931.822111] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9931.892098] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9936.124238] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 9941.992953] LustreError: 305015:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9942.009918] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9942.029165] LustreError: 305015:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 9946.821715] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9947.659380] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9952.752354] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9955.034812] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 9962.471550] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9965.092573] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9968.942696] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9970.413867] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9977.841206] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9977.848754] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9977.865618] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9977.886663] Lustre: Skipped 2 previous similar messages [ 9981.517172] Lustre: server umount lustre-OST0000 complete [ 9988.582262] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9988.590326] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9988.600653] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9988.600664] Lustre: Skipped 2 previous similar messages [ 9988.608355] Lustre: Skipped 2 previous similar messages [ 9988.636833] LustreError: Skipped 1 previous similar message [10000.354626] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10000.519712] Lustre: server umount lustre-MDT0000 complete [10003.938439] LustreError: 305013:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10003.947779] LustreError: 305013:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10004.564826] LustreError: 301720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781238078 with bad export cookie 383247876642597420 [10004.567314] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10004.576129] LustreError: 301720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10004.960537] Lustre: server umount lustre-MDT0001 complete [10015.434985] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [10018.405459] Key type lgssc unregistered [10018.768818] LNet: 307443:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10018.775393] LNetError: 307443:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10018.788972] LNet: Removed LNI 192.168.202.154@tcp [10019.728974] Key type .llcrypt unregistered [10019.731849] Key type ._llcrypt unregistered [10045.507313] Key type ._llcrypt registered [10045.521497] Key type .llcrypt registered [10045.713782] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [10047.255182] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10047.320892] alg: No test for adler32 (adler32-zlib) [10048.432849] Lustre: Lustre: Build Version: 2.17.53_64_gab2e0dd [10048.737832] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [10050.376160] Key type lgssc registered [10051.488316] Lustre: Echo OBD driver; http://www.lustre.org/ [10059.038479] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10059.055086] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10060.674346] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10066.053221] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [10071.011392] LustreError: 308853:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10071.038194] LustreError: 308853:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10075.814202] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10076.134148] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [10076.321084] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10081.120175] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [10088.370218] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10093.216248] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10096.610878] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10096.617377] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10096.635428] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10097.121705] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10101.217829] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10101.224147] Lustre: Skipped 3 previous similar messages [10102.480222] Lustre: server umount lustre-MDT0000 complete [10106.071404] LustreError: 308833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781238180 with bad export cookie 18010373199874421472 [10106.074151] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10106.085854] LustreError: 308833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10106.434045] Lustre: server umount lustre-MDT0001 complete [10122.564904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10123.008694] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10123.045901] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10123.255541] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10128.704979] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [10137.509236] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10137.590840] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10137.617663] Lustre: 311217:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10137.843170] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10142.126667] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [10147.932688] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10153.072500] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10164.216246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10164.501761] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10164.509708] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10164.521333] LustreError: 312875:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [10164.534420] LustreError: 312875:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [10164.538817] LustreError: 312875:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10164.657699] Lustre: server umount lustre-OST0000 complete [10164.662792] LustreError: 312875:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [10176.430098] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10176.700099] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10176.917283] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10183.471473] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [10186.415637] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10189.761735] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10197.218841] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10197.346671] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10197.463693] Lustre: lustre-OST0001: new disk, initializing [10197.471430] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10197.482882] Lustre: 314305:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10197.598570] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10204.552850] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [10206.864578] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10206.873741] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10207.034875] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10210.516055] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [10217.467872] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10217.469632] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10217.477046] LustreError: Skipped 1 previous similar message [10217.480776] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10217.501777] Lustre: Skipped 1 previous similar message [10222.588426] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10222.598844] Lustre: Skipped 2 previous similar messages [10228.192219] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10228.654631] Lustre: server umount lustre-OST0001 complete [10232.808893] LustreError: 313341:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10232.825681] LustreError: 313341:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10237.925606] LustreError: 313342:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10237.935215] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10237.949769] LustreError: 313342:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10237.959822] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10237.961581] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10238.013854] Lustre: Skipped 3 previous similar messages [10239.942340] Lustre: server umount lustre-OST0000 complete [10245.601077] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10245.608099] LustreError: Skipped 1 previous similar message [10245.612120] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10245.621524] Lustre: Skipped 1 previous similar message [10250.889080] Lustre: server umount lustre-MDT0000 complete [10253.281626] LustreError: 313627:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10253.310503] LustreError: 313627:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10255.421790] LustreError: 311939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781238329 with bad export cookie 18010373199874422025 [10255.422269] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10255.439677] LustreError: 311939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10255.709306] Lustre: server umount lustre-MDT0001 complete [10267.467718] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [10271.437350] Key type lgssc unregistered [10271.701647] LNet: 316136:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10271.709655] LNetError: 316136:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10271.757732] LNet: Removed LNI 192.168.202.154@tcp [10272.493370] Key type .llcrypt unregistered [10272.496372] Key type ._llcrypt unregistered [10353.889675] Key type ._llcrypt registered [10353.892337] Key type .llcrypt registered [10354.026742] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [10371.627692] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [10373.289313] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10373.417754] alg: No test for adler32 (adler32-zlib) [10374.649409] Lustre: Lustre: Build Version: 2.17.53_64_gab2e0dd [10374.916865] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [10376.687203] Key type lgssc registered [10378.072236] Lustre: Echo OBD driver; http://www.lustre.org/ [10423.066333] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10423.087822] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10424.623878] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10424.653956] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10424.834210] Lustre: lustre-MDT0000: new disk, initializing [10424.954859] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10424.964914] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10431.183185] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [10442.598182] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10442.678535] Lustre: 321038:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10442.711442] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10442.714548] Lustre: Skipped 1 previous similar message [10442.820175] Lustre: lustre-MDT0001: new disk, initializing [10442.887176] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10442.929111] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10442.936873] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10446.560724] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [10455.354747] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10460.134167] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10467.179926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10467.533753] Lustre: lustre-OST0000: new disk, initializing [10467.537379] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10467.545354] Lustre: 322782:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10467.625634] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10473.932269] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [10474.058415] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10474.076113] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10474.206333] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10481.769603] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10486.310714] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10486.525093] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10490.512126] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10490.764776] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10494.944704] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10494.952645] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10494.971166] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10498.847407] Lustre: server umount lustre-OST0000 complete [10504.678830] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10504.690215] LustreError: Skipped 1 previous similar message [10504.692446] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10504.698881] Lustre: Skipped 1 previous similar message [10504.703445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10504.708784] Lustre: Skipped 1 previous similar message [10508.997275] Lustre: server umount lustre-MDT0000 complete [10512.773044] LustreError: 321029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781238586 with bad export cookie 15923360798950503146 [10512.779125] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10512.783471] LustreError: 321029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10513.132125] Lustre: server umount lustre-MDT0001 complete [10521.916903] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 00:29:54 (1781238594) [10529.904699] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 00:30:02 (1781238602) [10539.404793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10539.754366] LustreError: 325389:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10539.763502] LustreError: 325389:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10539.827504] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10543.846248] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [10545.122855] LustreError: 325390:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10549.218359] LustreError: 325389:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10551.794818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10552.051268] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10555.797495] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [10561.887265] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10565.687958] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10572.334199] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10572.733528] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10578.829498] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [10584.356870] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10593.506742] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10596.101830] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10598.704201] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10601.466113] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10604.207431] Lustre: Failing over lustre-MDT0000 [10605.052181] Lustre: server umount lustre-MDT0000 complete [10605.537960] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10605.551045] LustreError: 325386:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10605.565675] Lustre: Skipped 3 previous similar messages [10605.610687] LustreError: 325386:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10610.672908] LustreError: 327000:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10610.700225] LustreError: 327000:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10620.926546] LustreError: 325390:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10620.956169] LustreError: 325390:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [10621.920122] Lustre: 319261:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781238679/real 1781238679] req@00000000ace122ed x1867763886917888/t0(0) o400->MGC192.168.202.154@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1781238695 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10621.960691] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10626.699738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10631.141452] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xdcfb2447f3d33a76 [10631.150323] LustreError: 328253:0:(ldlm_resource.c:1180:ldlm_resource_complain()) MGC192.168.202.154@tcp: namespace resource [0x65727473756c:0x5:0x0].0x0 (00000000c697c57c) refcount nonzero (1) after lock cleanup; forcing cleanup. [10631.175491] Lustre: MGC192.168.202.154@tcp: Connection restored to 0@lo (at 0@lo) [10631.483680] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10631.554230] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10634.224518] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10636.797035] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10636.842779] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [10636.911923] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10637.679553] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [10647.193148] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10649.192154] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10653.578251] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10655.668389] Lustre: Failing over lustre-MDT0000 [10655.913796] Lustre: server umount lustre-MDT0000 complete [10657.258213] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10657.264840] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10657.279211] LustreError: 325389:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10657.290909] Lustre: Skipped 2 previous similar messages [10657.339194] LustreError: 325389:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 16 previous similar messages [10673.632115] Lustre: 319260:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781238731/real 1781238731] req@000000002a198454 x1867763886939392/t0(0) o400->MGC192.168.202.154@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1781238747 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10673.676657] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10677.178691] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10684.306937] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10684.408309] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10686.429050] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10689.548057] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10689.569157] Lustre: Skipped 2 previous similar messages [10689.633994] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [10689.742428] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10690.258512] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [10699.645815] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10701.658632] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10709.990472] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10709.995874] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10710.020362] Lustre: Skipped 2 previous similar messages [10710.031544] Lustre: Skipped 4 previous similar messages [10714.307643] Lustre: server umount lustre-MDT0000 complete [10715.112287] LustreError: 326099:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10715.152981] LustreError: 326099:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 26 previous similar messages [10718.449908] LustreError: 329852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781238792 with bad export cookie 15923360798950505869 [10718.465399] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10718.466808] LustreError: 329852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10718.710919] Lustre: server umount lustre-MDT0001 complete [10734.050899] Lustre: server umount lustre-OST0000 complete [10750.581036] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10751.398110] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10756.312775] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [10766.478326] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10772.219949] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [10779.619603] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10784.106959] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10792.005170] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10796.404131] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10799.769525] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [10806.469280] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10822.118334] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10822.123161] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10822.131121] LustreError: Skipped 1 previous similar message [10822.133775] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10822.183379] Lustre: Skipped 1 previous similar message [10824.464873] Lustre: server umount lustre-OST0000 complete [10832.364728] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10832.368582] Lustre: Skipped 2 previous similar messages [10835.539423] Lustre: server umount lustre-MDT0000 complete [10837.478261] LustreError: 331163:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10837.514600] LustreError: 331163:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [10840.977745] LustreError: 332786:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781238914 with bad export cookie 15923360798950506618 [10840.979950] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10840.995909] LustreError: 332786:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10841.594655] Lustre: server umount lustre-MDT0001 complete [10854.285350] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [10858.106151] Key type lgssc unregistered [10858.489867] LNet: 334347:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10858.501551] LNetError: 334347:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10858.513972] LNet: Removed LNI 192.168.202.154@tcp [10859.627488] Key type .llcrypt unregistered [10859.629422] Key type ._llcrypt unregistered [10888.836644] Key type ._llcrypt registered [10888.840267] Key type .llcrypt registered [10888.964773] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [10890.013069] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10890.154135] alg: No test for adler32 (adler32-zlib) [10891.375447] Lustre: Lustre: Build Version: 2.17.53_64_gab2e0dd [10891.606722] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [10893.255218] Key type lgssc registered [10894.303200] Lustre: Echo OBD driver; http://www.lustre.org/ [10903.606576] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10903.642557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10905.317539] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10909.977498] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [10915.814188] LustreError: 335850:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10915.841463] LustreError: 335850:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10919.698364] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10920.205731] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10925.572479] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [10932.770935] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10938.221822] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10947.725818] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10948.202635] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10950.301847] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10956.444549] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [10963.383100] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10981.344351] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10981.347018] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10981.358196] Lustre: Skipped 2 previous similar messages [10981.362352] Lustre: Skipped 1 previous similar message [10983.774764] Lustre: server umount lustre-MDT0000 complete [10986.470450] LustreError: 337163:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10986.490921] LustreError: 337163:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10987.922561] LustreError: 335828:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781239061 with bad export cookie 11993826740067854896 [10987.938881] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10987.942045] LustreError: 335828:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10988.311733] Lustre: server umount lustre-MDT0001 complete [11003.074232] Lustre: server umount lustre-OST0000 complete [11022.102421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11022.649766] LustreError: 339246:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11022.765463] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11027.939253] LustreError: 339247:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11028.998152] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [11032.037408] LustreError: 339246:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11041.010518] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11041.383561] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11046.125585] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [11054.604952] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11060.392353] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11068.056650] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11068.446940] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11075.450870] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [11082.492431] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11092.997954] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11107.808569] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11107.817822] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11107.838126] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11107.846927] Lustre: Skipped 1 previous similar message [11113.450381] LustreError: 340271:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11113.475973] LustreError: 340271:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11113.541411] Lustre: server umount lustre-MDT0000 complete [11118.123595] LustreError: 339227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781239192 with bad export cookie 11993826740067856009 [11118.133142] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11118.134170] LustreError: 339227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11118.563023] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11118.581671] Lustre: Skipped 2 previous similar messages [11118.806343] Lustre: server umount lustre-MDT0001 complete [11126.004969] Lustre: server umount lustre-OST0000 complete [11147.578808] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11148.210608] LustreError: 342839:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11148.216959] LustreError: 342839:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11148.273417] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11153.623453] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [11166.478464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11172.730962] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [11179.858611] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11184.729983] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11193.670624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11193.895080] LustreError: 344436:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11193.918090] LustreError: 344436:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11194.122170] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11194.135743] Lustre: Skipped 1 previous similar message [11199.488222] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11201.878737] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [11209.601926] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11217.889387] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11217.894710] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11217.901105] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11217.916364] Lustre: Skipped 1 previous similar message [11217.920916] Lustre: Skipped 3 previous similar messages [11222.468651] Lustre: server umount lustre-OST0000 complete [11228.128907] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11228.146693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11228.156097] Lustre: Skipped 2 previous similar messages [11232.686792] Lustre: server umount lustre-MDT0000 complete [11237.143388] LustreError: 345282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781239311 with bad export cookie 11993826740067857031 [11237.153427] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11237.157629] LustreError: 345282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11237.612414] Lustre: server umount lustre-MDT0001 complete [11250.020263] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [11252.701749] Key type lgssc unregistered [11252.937915] LNet: 346018:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11252.947420] LNetError: 346018:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11253.992543] LNet: Removed LNI 192.168.202.154@tcp [11254.965144] Key type .llcrypt unregistered [11254.967599] Key type ._llcrypt unregistered [11284.481763] Key type ._llcrypt registered [11284.488809] Key type .llcrypt registered [11284.665824] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [11286.166774] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11286.254825] alg: No test for adler32 (adler32-zlib) [11287.383453] Lustre: Lustre: Build Version: 2.17.53_64_gab2e0dd [11287.742336] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [11289.471344] Key type lgssc registered [11290.858408] Lustre: Echo OBD driver; http://www.lustre.org/ [11300.437888] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11300.474181] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11300.483401] LustreError: 347450:0:(osd_handler.c:8551:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11300.505441] LustreError: 347450:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11300.511085] LustreError: 347450:0:(obd_mount.c:250:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11300.518279] LustreError: 347450:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11300.527797] LustreError: 347450:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [11309.603045] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11309.610150] LustreError: 347847:0:(osd_handler.c:8551:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11309.655703] LustreError: 347847:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11309.674692] LustreError: 347847:0:(obd_mount.c:250:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11309.692798] LustreError: 347847:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11309.707364] LustreError: 347847:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [11332.463241] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [11335.536414] Key type lgssc unregistered [11335.846070] LNet: 348834:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11335.854966] LNetError: 348834:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11335.874916] LNet: Removed LNI 192.168.202.154@tcp [11336.822664] Key type .llcrypt unregistered [11336.827130] Key type ._llcrypt unregistered [11359.470388] Key type ._llcrypt registered [11359.475119] Key type .llcrypt registered [11359.682589] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [11378.651672] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [11379.888436] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11379.975109] alg: No test for adler32 (adler32-zlib) [11381.366512] Lustre: Lustre: Build Version: 2.17.53_64_gab2e0dd [11381.660716] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [11383.297299] Key type lgssc registered [11384.537343] Lustre: Echo OBD driver; http://www.lustre.org/ [11428.254886] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11428.286035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11429.571578] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11429.598905] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11429.711389] Lustre: lustre-MDT0000: new disk, initializing [11429.802537] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11429.832263] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11434.957601] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [11445.954551] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11446.122754] Lustre: 351962:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11446.173082] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11446.180516] Lustre: Skipped 1 previous similar message [11446.255730] Lustre: lustre-MDT0001: new disk, initializing [11446.394924] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11446.436100] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11446.443201] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11450.932571] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [11459.386990] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11463.392347] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11471.348902] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11471.579452] Lustre: lustre-OST0000: new disk, initializing [11471.584221] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11471.590245] Lustre: 353703:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11471.679566] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11477.053894] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11477.063506] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11477.195826] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11478.308929] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [11487.232994] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11492.362456] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11492.653341] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11498.013107] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11498.281394] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11503.075529] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11503.084168] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11503.092384] LustreError: Skipped 1 previous similar message [11503.094030] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11503.122028] Lustre: Skipped 1 previous similar message [11508.217958] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11508.233742] Lustre: Skipped 1 previous similar message [11513.328093] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11513.343362] Lustre: Skipped 1 previous similar message [11515.359185] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11515.664870] Lustre: server umount lustre-OST0000 complete [11523.043474] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11523.054900] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11523.067263] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11523.069715] Lustre: Skipped 1 previous similar message [11526.508577] Lustre: server umount lustre-MDT0000 complete [11530.298836] LustreError: 352740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781239604 with bad export cookie 2978438488645530518 [11530.303941] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11530.306957] LustreError: 352740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11530.517945] Lustre: server umount lustre-MDT0001 complete [11543.255747] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11545.201047] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 00:46:57 (1781239617) [11552.172648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11552.690348] LustreError: 356016:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11552.712852] LustreError: 356016:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11552.777137] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11557.861896] LustreError: 356017:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11558.061407] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [11562.979711] LustreError: 356016:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11567.740409] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11568.054843] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11573.029452] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [11579.330557] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11584.200218] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11592.075901] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11592.465142] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11599.566283] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [11605.053943] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11613.021700] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11613.390880] Lustre: lustre-OST0001: new disk, initializing [11613.403157] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11613.426397] Lustre: 358513:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11613.601549] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11620.982830] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [11622.520672] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11622.539697] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11622.635369] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11628.864745] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [11637.222744] Lustre: lustre-OST0001: Not available for connect from 192.168.202.54@tcp (stopping) [11637.244666] Lustre: Skipped 1 previous similar message [11638.247804] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11638.248670] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11638.256439] LustreError: Skipped 1 previous similar message [11638.286286] Lustre: Skipped 2 previous similar messages [11640.425455] Lustre: server umount lustre-OST0001 complete [11642.362101] LustreError: 358943:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11648.485166] LustreError: 357611:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11648.505211] LustreError: 357611:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11658.752571] LustreError: 357612:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11658.804337] LustreError: 357612:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11660.261044] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11660.264936] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11660.280580] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11660.283269] Lustre: Skipped 2 previous similar messages [11666.617964] Lustre: server umount lustre-OST0000 complete [11674.084250] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11674.103916] Lustre: Skipped 2 previous similar messages [11677.634524] Lustre: server umount lustre-MDT0000 complete [11679.202424] LustreError: 356016:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11679.210994] LustreError: 356016:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11682.124835] LustreError: 355998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781239756 with bad export cookie 2978438488645531400 [11682.130379] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11682.147587] LustreError: 355998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11688.857594] Lustre: server umount lustre-MDT0001 complete [11701.827542] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [11705.776770] Key type lgssc unregistered [11706.163092] LNet: 360371:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11706.173749] LNetError: 360371:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11706.187104] LNet: Removed LNI 192.168.202.154@tcp [11707.496801] Key type .llcrypt unregistered [11707.512052] Key type ._llcrypt unregistered [11767.810499] Key type ._llcrypt registered [11767.812813] Key type .llcrypt registered [11768.003302] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [11770.272596] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11770.338170] alg: No test for adler32 (adler32-zlib) [11771.668423] Lustre: Lustre: Build Version: 2.17.53_64_gab2e0dd [11772.009535] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [11773.863193] Key type lgssc registered [11775.347171] Lustre: Echo OBD driver; http://www.lustre.org/ [11784.717406] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11784.741400] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11786.058091] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11786.111566] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11786.280548] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11791.592602] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [11802.374906] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11802.692288] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11802.752399] Lustre: 363154:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11803.313580] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11809.816459] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [11818.482589] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11824.226953] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11827.173968] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11827.199179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11827.203984] Lustre: Skipped 1 previous similar message [11832.294205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11832.303823] Lustre: Skipped 1 previous similar message [11833.697049] Lustre: server umount lustre-MDT0000 complete [11838.658321] LustreError: 364523:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781239912 with bad export cookie 4703144318542684205 [11838.688975] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11838.689396] LustreError: 364523:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11839.406499] Lustre: server umount lustre-MDT0001 complete [11851.512282] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 00:52:03 (1781239923) [11859.179930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11859.863253] LustreError: 365444:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11859.896578] LustreError: 365444:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11859.968019] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11865.063792] LustreError: 365445:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11865.462678] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [11870.178038] LustreError: 365444:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11875.317335] LustreError: 365445:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11875.420850] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11875.955765] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11881.467237] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [11888.752540] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11893.588859] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11902.089942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11902.226192] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11902.421418] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11910.202218] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [11917.220928] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11926.577825] Lustre: Permanently deactivating lustre-OST0000 [11926.603070] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11932.741928] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11935.026729] LustreError: 368046:0:(mgs_llog.c:2031:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11935.045713] LustreError: 368046:0:(mgs_llog.c:2086:mgs_replace_nids()) Only MGS is allowed to be started [11935.056504] LustreError: 368046:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11938.659532] LustreError: 368095:0:(mgs_llog.c:2086:mgs_replace_nids()) Only MGS is allowed to be started [11938.667827] LustreError: 368095:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11943.496477] Lustre: server umount lustre-OST0000 complete [11945.949923] LustreError: 368345:0:(mgs_llog.c:2086:mgs_replace_nids()) Only MGS is allowed to be started [11945.956638] LustreError: 368345:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11952.615926] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11952.617101] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11952.622472] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11952.630931] Lustre: Skipped 2 previous similar messages [11957.728890] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11957.747198] Lustre: Skipped 2 previous similar messages [11962.335756] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11962.636701] Lustre: server umount lustre-MDT0000 complete [11962.854286] LustreError: 366391:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11967.500325] LustreError: 365426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781240041 with bad export cookie 4703144318542684786 [11967.510156] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11967.522542] LustreError: 365426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [11977.994607] Lustre: server umount lustre-MDT0001 complete [11988.540986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11993.684557] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [11996.937890] LustreError: 369487:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [12003.974409] Lustre: 369827:0:(mgs_llog.c:1798:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.154@tcp [12006.731370] Lustre: server umount MGS complete [12019.580305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12019.928982] LustreError: 370552:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12019.960433] LustreError: 370552:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12019.969296] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12019.981268] Lustre: Skipped 1 previous similar message [12020.051890] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12025.701335] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [12036.511263] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12036.946759] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12037.002954] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12041.960864] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [12049.400921] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12054.197492] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12056.523234] Lustre: Permanently reactivating lustre-OST0000 [12056.555176] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12056.562514] Lustre: Skipped 2 previous similar messages [12086.752091] LustreError: 372154:0:(osp_dev.c:693:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12094.457266] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12094.877208] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12101.849457] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [12108.889921] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12116.452106] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12116.479756] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12116.509435] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12126.708105] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12126.717020] Lustre: Skipped 3 previous similar messages [12130.273698] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12130.605231] Lustre: server umount lustre-OST0000 complete [12136.935803] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12136.950415] Lustre: Skipped 2 previous similar messages [12141.653710] Lustre: server umount lustre-MDT0000 complete [12142.051426] LustreError: 371511:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12142.069986] LustreError: 371511:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [12146.804071] LustreError: 370534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781240220 with bad export cookie 4703144318542685836 [12146.821051] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12147.445113] Lustre: server umount lustre-MDT0001 complete [12161.104946] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [12164.030446] Key type lgssc unregistered [12164.451630] LNet: 374562:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12164.456826] LNetError: 374562:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12164.470367] LNet: Removed LNI 192.168.202.154@tcp [12165.339523] Key type .llcrypt unregistered [12165.344607] Key type ._llcrypt unregistered [12187.422324] Key type ._llcrypt registered [12187.424607] Key type .llcrypt registered [12187.614648] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [12207.941312] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [12209.269771] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12209.458985] alg: No test for adler32 (adler32-zlib) [12210.522312] Lustre: Lustre: Build Version: 2.17.53_64_gab2e0dd [12210.741823] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [12212.432888] Key type lgssc registered [12214.118240] Lustre: Echo OBD driver; http://www.lustre.org/ [12250.829901] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12250.845044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12252.157505] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12252.202454] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12252.256542] Lustre: lustre-MDT0000: new disk, initializing [12252.328487] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12252.343130] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12257.195994] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [12267.747370] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12267.889878] Lustre: 377686:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12267.949554] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12267.953349] Lustre: Skipped 1 previous similar message [12268.080908] Lustre: lustre-MDT0001: new disk, initializing [12268.206871] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12268.275771] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12268.299563] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12273.409913] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [12281.424408] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12284.894548] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12291.961983] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12292.318382] Lustre: lustre-OST0000: new disk, initializing [12292.321474] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12292.325335] Lustre: 379429:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12292.420176] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12298.044369] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [12298.401299] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12298.420869] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12298.586462] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12307.383979] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12312.930683] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12313.216291] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12319.062334] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12319.369623] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12324.323803] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12324.326169] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12324.329764] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12324.340216] Lustre: Skipped 1 previous similar message [12324.358758] Lustre: Skipped 1 previous similar message [12327.370849] Lustre: server umount lustre-OST0000 complete [12334.561315] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12334.572936] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12334.579132] Lustre: Skipped 1 previous similar message [12334.599213] Lustre: Skipped 1 previous similar message [12337.812795] Lustre: server umount lustre-MDT0000 complete [12341.123417] LustreError: 377680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781240415 with bad export cookie 3250226084266653616 [12341.137435] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12341.362118] Lustre: server umount lustre-MDT0001 complete [12349.640972] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12351.357922] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 01:00:23 (1781240423) [12359.244699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12360.089163] LustreError: 381742:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12360.115240] LustreError: 381742:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [12360.203274] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12365.287376] LustreError: 381743:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12365.516867] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [12370.400212] LustreError: 381742:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12375.528254] LustreError: 381743:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12376.027412] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12376.435836] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12382.921927] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [12390.230933] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12394.462646] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12400.753385] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12400.991879] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12407.475809] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [12413.365768] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12415.162730] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12417.654679] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12422.625400] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12422.650169] LustreError: Skipped 1 previous similar message [12422.652398] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12422.690310] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12425.699961] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12425.703518] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12425.723262] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12428.457747] Lustre: server umount lustre-OST0000 complete [12432.864402] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12432.869528] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12432.881928] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12432.886057] Lustre: Skipped 1 previous similar message [12438.858868] Lustre: server umount lustre-MDT0000 complete [12441.067554] LustreError: 384052:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12441.088452] LustreError: 384052:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12443.159535] LustreError: 382460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781240517 with bad export cookie 3250226084266654498 [12443.165300] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12443.179243] LustreError: 382460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [12443.542061] Lustre: server umount lustre-MDT0001 complete [12455.438446] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [12458.395536] Key type lgssc unregistered [12458.717745] LNet: 385124:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12458.737279] LNetError: 385124:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12459.815687] LNet: Removed LNI 192.168.202.154@tcp [12460.811368] Key type .llcrypt unregistered [12460.813568] Key type ._llcrypt unregistered [12486.796943] Key type ._llcrypt registered [12486.798788] Key type .llcrypt registered [12486.899652] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [12487.136562] Key type .llcrypt unregistered [12487.138852] Key type ._llcrypt unregistered [12507.048694] Key type ._llcrypt registered [12507.052099] Key type .llcrypt registered [12507.152338] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [12508.197619] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12508.282053] alg: No test for adler32 (adler32-zlib) [12509.324245] Lustre: Lustre: Build Version: 2.17.53_64_gab2e0dd [12509.547585] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [12511.248132] Key type lgssc registered [12512.413479] Lustre: Echo OBD driver; http://www.lustre.org/ [12520.148630] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12520.173265] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12521.641593] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12526.114857] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [12531.169716] LustreError: 387305:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12531.190611] LustreError: 387305:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12536.292254] LustreError: 387304:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12536.630056] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12537.021662] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12542.444634] LustreError: 387305:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12542.453122] LustreError: 387305:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12544.738634] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [12546.533306] LustreError: 387304:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12546.562300] LustreError: 387304:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12546.563133] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12551.650049] LustreError: 387305:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12551.660102] LustreError: 387305:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12552.521578] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12560.394485] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12560.825750] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12565.036200] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [12576.053829] Lustre: server umount lustre-OST0000 complete [12580.861900] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [12580.873765] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12580.907245] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12586.467790] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12586.484386] Lustre: Skipped 3 previous similar messages [12590.694723] Lustre: server umount lustre-MDT0000 complete [12591.589846] LustreError: 388910:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12591.623266] LustreError: 388910:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [12592.607159] Lustre: 386613:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781240650/real 1781240650] req@000000005a8e7bb2 x1867766125587072/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1781240666 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12592.628261] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12592.637054] Lustre: Skipped 1 previous similar message [12594.932799] LustreError: 387285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781240668 with bad export cookie 7522447343210589938 [12594.950209] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12595.268742] Lustre: server umount lustre-MDT0001 complete [12607.250190] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [12610.026254] Key type lgssc unregistered [12610.299156] LNet: 390287:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12610.306751] LNetError: 390287:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12610.319202] LNet: Removed LNI 192.168.202.154@tcp [12611.139817] Key type .llcrypt unregistered [12611.141821] Key type ._llcrypt unregistered [12637.890365] Key type ._llcrypt registered [12637.892922] Key type .llcrypt registered [12638.025455] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [12639.239150] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12639.418904] alg: No test for adler32 (adler32-zlib) [12640.490582] Lustre: Lustre: Build Version: 2.17.53_64_gab2e0dd [12640.657281] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [12642.303293] Key type lgssc registered [12643.602466] Lustre: Echo OBD driver; http://www.lustre.org/ [12652.037816] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12652.071159] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12667.679523] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12672.799399] LustreError: 391662:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.154@tcp: failed processing log, type 4: rc = -110 [12699.487360] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12707.413833] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [12714.631312] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12722.810747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12723.322926] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12723.417723] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12728.346915] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [12728.805033] LustreError: 391688:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12728.824155] LustreError: 391688:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12733.922482] LustreError: 391687:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12734.952219] LustreError: 392586:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12734.971904] LustreError: 392586:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12738.384800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12738.921336] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12744.981986] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [12751.780666] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12757.011345] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12767.066421] Lustre: server umount lustre-OST0000 complete [12771.326410] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [12771.340671] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12771.365884] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12774.895278] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12774.899885] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12780.021116] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12780.027919] Lustre: Skipped 2 previous similar messages [12781.213025] Lustre: server umount lustre-MDT0000 complete [12785.130907] LustreError: 392591:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12785.138663] Lustre: 390999:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781240843/real 1781240843] req@00000000a8d9f7d9 x1867766262953728/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1781240859 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12785.152936] LustreError: 392591:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12785.211513] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12785.604479] LustreError: 391670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781240859 with bad export cookie 17996842128169397102 [12785.620416] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12785.960841] Lustre: server umount lustre-MDT0001 complete [12799.356414] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [12802.883608] Key type lgssc unregistered [12803.296647] LNet: 394875:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12803.300493] LNetError: 394875:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12804.327662] LNet: Removed LNI 192.168.202.154@tcp [12805.713074] Key type .llcrypt unregistered [12805.717668] Key type ._llcrypt unregistered [12832.907447] Key type ._llcrypt registered [12832.912546] Key type .llcrypt registered [12833.067213] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [12834.415746] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12834.622342] alg: No test for adler32 (adler32-zlib) [12835.627495] Lustre: Lustre: Build Version: 2.17.53_64_gab2e0dd [12835.956561] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [12837.743304] Key type lgssc registered [12839.430791] Lustre: Echo OBD driver; http://www.lustre.org/ [12849.889989] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12849.908942] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12851.686943] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12857.190592] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [12861.922343] LustreError: 396277:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12861.943624] LustreError: 396277:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12867.052838] LustreError: 396278:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12867.452088] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12867.705474] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12872.099377] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [12878.287281] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12882.226338] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12889.737544] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12890.237113] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12896.373243] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12896.826483] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [12902.971996] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12908.512380] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12908.526200] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12908.548555] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12910.568837] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.54@tcp (stopping) [12912.097403] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12912.099935] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12912.114433] Lustre: Skipped 1 previous similar message [12914.079852] Lustre: server umount lustre-MDT0000 complete [12915.675738] LustreError: 396273:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12916.713550] LustreError: 396258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.54@tcp arrived at 1781240990 with bad export cookie 2651660260429970146 [12916.725355] LustreError: 396258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12918.583670] LustreError: 396257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781240992 with bad export cookie 2651660260429969250 [12918.594440] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12918.612178] LustreError: 396257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12922.336631] LustreError: 397266:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12922.345949] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12922.352259] LustreError: 397266:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [12922.386623] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12922.399190] Lustre: Skipped 2 previous similar messages [12924.876675] Lustre: server umount lustre-OST0000 complete [12928.500387] LustreError: 396278:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12928.522616] LustreError: 396278:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [12931.525366] Lustre: server umount lustre-MDT0001 complete [12940.665598] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [12943.361556] Key type lgssc unregistered [12943.690193] LNet: 399609:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12943.704444] LNetError: 399609:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12943.723169] LNet: Removed LNI 192.168.202.154@tcp [12944.332306] Key type .llcrypt unregistered [12944.334527] Key type ._llcrypt unregistered [12971.410506] Key type ._llcrypt registered [12971.417446] Key type .llcrypt registered [12971.563357] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [12973.224734] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12973.269639] alg: No test for adler32 (adler32-zlib) [12974.307214] Lustre: Lustre: Build Version: 2.17.53_64_gab2e0dd [12974.483542] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [12976.175184] Key type lgssc registered [12976.933213] Lustre: Echo OBD driver; http://www.lustre.org/ [12984.851233] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12984.862540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12986.416735] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12991.001199] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [12996.583750] LustreError: 401014:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12996.609541] LustreError: 401014:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12999.844763] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13000.098818] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13005.137798] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [13012.976202] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13016.983482] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13023.963191] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13024.516095] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13031.308920] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [13031.674309] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [13036.178783] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13041.130681] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13041.138555] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13041.152764] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13042.150282] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13042.153484] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13044.229939] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.54@tcp (stopping) [13044.239295] Lustre: Skipped 1 previous similar message [13047.264909] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13047.270318] Lustre: Skipped 1 previous similar message [13047.369890] Lustre: server umount lustre-MDT0001 complete [13052.387589] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13052.388934] LustreError: 401014:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13052.401687] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13052.405664] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13057.530433] LustreError: 402661:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13058.832869] Lustre: server umount lustre-OST0000 complete [13062.638851] LustreError: 401014:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13063.352551] Lustre: server umount lustre-MDT0000 complete [13078.484256] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [13081.421742] Key type lgssc unregistered [13081.879559] LNet: 404341:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13081.891524] LNetError: 404341:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13081.909838] LNet: Removed LNI 192.168.202.154@tcp [13082.925718] Key type .llcrypt unregistered [13082.931452] Key type ._llcrypt unregistered [13114.704448] Key type ._llcrypt registered [13114.706447] Key type .llcrypt registered [13114.833473] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [13131.119890] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [13132.847783] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13132.866351] alg: No test for adler32 (adler32-zlib) [13134.026469] Lustre: Lustre: Build Version: 2.17.53_64_gab2e0dd [13134.367673] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [13136.087194] Key type lgssc registered [13137.318973] Lustre: Echo OBD driver; http://www.lustre.org/ [13176.999112] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [13193.227476] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13193.263951] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13194.590384] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13194.625415] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13194.732980] Lustre: lustre-MDT0000: new disk, initializing [13194.814506] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13194.831123] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13198.868286] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [13214.707616] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13215.073598] Lustre: lustre-OST0000: new disk, initializing [13215.076818] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13215.080576] Lustre: Skipped 1 previous similar message [13215.087693] Lustre: 409308:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13215.155684] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13222.203873] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [13225.121179] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13225.131412] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13225.160914] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13230.424170] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13248.376209] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13248.522738] Lustre: 408309:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13248.580981] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13248.642829] Lustre: lustre-MDT0001: new disk, initializing [13248.756887] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13248.822563] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13248.837536] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13253.574982] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [13262.638542] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13262.855992] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13267.495690] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13267.714223] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13273.058915] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13273.059890] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13273.064102] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13273.105736] Lustre: Skipped 1 previous similar message [13275.628625] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.54@tcp (stopping) [13275.634899] Lustre: Skipped 1 previous similar message [13278.187955] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13278.199031] Lustre: Skipped 1 previous similar message [13280.731599] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.54@tcp (stopping) [13285.835995] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.54@tcp (stopping) [13285.843794] Lustre: Skipped 2 previous similar messages [13287.391834] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13287.615808] Lustre: server umount lustre-MDT0001 complete [13293.549955] LustreError: 408319:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13293.577979] LustreError: 408319:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [13298.664525] LustreError: 408315:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13298.686323] LustreError: 408315:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13303.799751] LustreError: 409315:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13303.819630] LustreError: 409315:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13308.898036] LustreError: 408320:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13308.900076] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13308.959923] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13313.903780] Lustre: server umount lustre-OST0000 complete [13314.016833] LustreError: 408319:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13324.278670] Lustre: server umount lustre-MDT0000 complete [13336.902923] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [13339.382812] Key type lgssc unregistered [13339.697596] LNet: 412883:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13339.709446] LNetError: 412883:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13339.729276] LNet: Removed LNI 192.168.202.154@tcp [13340.566331] Key type .llcrypt unregistered [13340.568577] Key type ._llcrypt unregistered [13381.345403] Key type ._llcrypt registered [13381.348803] Key type .llcrypt registered [13381.416951] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [13381.606421] Key type .llcrypt unregistered [13381.608817] Key type ._llcrypt unregistered [13399.207603] Key type ._llcrypt registered [13399.209555] Key type .llcrypt registered [13399.377377] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [13400.720383] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13400.784761] alg: No test for adler32 (adler32-zlib) [13402.108748] Lustre: Lustre: Build Version: 2.17.53_64_gab2e0dd [13402.460182] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [13404.223209] Key type lgssc registered [13405.497448] Lustre: Echo OBD driver; http://www.lustre.org/ [13435.342478] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13435.383118] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13436.676879] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13436.733699] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13436.842762] Lustre: lustre-MDT0000: new disk, initializing [13436.966372] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13436.986332] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13441.677094] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [13453.732203] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13453.882231] Lustre: 416224:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13453.917416] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13453.919807] Lustre: Skipped 1 previous similar message [13453.977973] Lustre: lustre-MDT0001: new disk, initializing [13454.084812] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13454.113499] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13454.131768] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13459.085540] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [13467.393975] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13472.056631] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13479.637825] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13480.000220] Lustre: lustre-OST0000: new disk, initializing [13480.004371] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13480.012471] Lustre: 417969:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13480.108673] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13486.655805] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [13488.226349] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13488.257653] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13488.367292] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13494.990636] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13503.985694] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13503.992239] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13503.997934] Lustre: Skipped 2 previous similar messages [13504.000519] Lustre: Skipped 1 previous similar message [13509.090053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13509.097783] Lustre: Skipped 3 previous similar messages [13509.261204] Lustre: server umount lustre-MDT0000 complete [13513.713627] LustreError: 416219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781241587 with bad export cookie 1388267778518324643 [13513.720063] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13513.729907] LustreError: 416219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13514.234239] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13514.460680] Lustre: server umount lustre-MDT0001 complete [13520.699417] Lustre: server umount lustre-OST0000 complete [13532.625076] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 01:20:04 (1781241604) [13542.223418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13542.745768] LustreError: 419881:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13542.768986] LustreError: 419881:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [13542.842343] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13548.011576] LustreError: 419882:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13548.172409] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [13552.108500] LustreError: 419881:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13557.218785] LustreError: 419882:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13558.230956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13558.598350] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13563.756319] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [13571.440191] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13575.817787] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13596.400785] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13596.617231] Lustre: Found index 0 for lustre-OST0000, updating log [13596.949546] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13603.609345] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [13610.982199] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13613.372327] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13624.800838] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13624.812785] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13624.834168] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13626.869633] Lustre: server umount lustre-OST0000 complete [13645.793789] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13645.800078] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13645.813863] Lustre: Skipped 1 previous similar message [13645.822036] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13645.836461] Lustre: Skipped 3 previous similar messages [13650.587426] Lustre: server umount lustre-MDT0000 complete [13652.453103] LustreError: 419882:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13655.437458] LustreError: 419862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781241729 with bad export cookie 1388267778518326036 [13655.442634] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13655.817084] Lustre: server umount lustre-MDT0001 complete [13664.585503] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 01:22:16 (1781241736) [13666.660879] LNet: 1 local NIs in recovery (showing 1): 192.168.202.154@tcp [14289.153138] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14289.568457] LustreError: 425971:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14289.584399] LustreError: 425971:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [14289.651819] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14295.013473] LustreError: 425972:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14295.254797] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [14299.107721] LustreError: 425971:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14305.263220] LustreError: 425972:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14305.290823] LustreError: 425972:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [14309.006461] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14309.600996] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14314.031585] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [14320.543208] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14324.890792] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14344.543318] LNet: 1 local NIs in recovery (showing 1): 192.168.202.154@tcp [14346.556658] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14347.008358] Lustre: Found index 0 for lustre-OST0000, updating log [14347.477913] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14355.368756] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [14361.880650] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14364.791462] Lustre: server umount lustre-OST0000 complete [14370.852058] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14370.867860] Lustre: Skipped 1 previous similar message [14370.878834] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14370.885215] Lustre: Skipped 3 previous similar messages [14371.938690] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14371.953538] Lustre: Skipped 1 previous similar message [14372.961028] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14372.973174] Lustre: Skipped 1 previous similar message [14375.009136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14375.020129] Lustre: Skipped 3 previous similar messages [14376.214718] Lustre: server umount lustre-MDT0000 complete [14377.066381] LustreError: 425971:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14377.096341] LustreError: 425971:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [14380.717916] LustreError: 425953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781242454 with bad export cookie 1388267778518327121 [14380.719244] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14380.738776] LustreError: 425953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14381.452392] Lustre: server umount lustre-MDT0001 complete [14403.999281] LNet: 1 local NIs in recovery (showing 1): 192.168.202.154@tcp [14414.629230] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [14425.772037] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [14470.919212] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 01:35:43 (1781242543) [14494.387228] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [14498.483763] Key type lgssc unregistered [14498.895771] LNet: 432197:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14498.911279] LNetError: 432197:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14607.471833] LNet: Removed LNI 192.168.202.154@tcp [14608.269144] Key type .llcrypt unregistered [14608.271523] Key type ._llcrypt unregistered [14627.163231] Key type ._llcrypt registered [14627.165400] Key type .llcrypt registered [14627.263487] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [14628.396341] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14628.520842] alg: No test for adler32 (adler32-zlib) [14629.814761] Lustre: Lustre: Build Version: 2.17.53_64_gab2e0dd [14630.260162] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [14631.999207] Key type lgssc registered [14633.078794] Lustre: Echo OBD driver; http://www.lustre.org/ [14637.990874] LNet: Added LNI 192.168.202.154@tcp100 [8/256/0/180] [14639.194810] LNet: Added LNI 192.168.202.154@tcp101 [8/256/0/180] [14640.301773] LNet: Added LNI 192.168.202.154@tcp102 [8/256/0/180] [14641.358574] LNet: Added LNI 192.168.202.154@tcp103 [8/256/0/180] [14642.343382] LNet: Added LNI 192.168.202.154@tcp104 [8/256/0/180] [14643.269596] LNet: Added LNI 192.168.202.154@tcp105 [8/256/0/180] [14644.179114] LNet: Added LNI 192.168.202.154@tcp106 [8/256/0/180] [14645.265243] LNet: Added LNI 192.168.202.154@tcp107 [8/256/0/180] [14646.507552] LNet: Added LNI 192.168.202.154@tcp108 [8/256/0/180] [14647.470292] LNet: Added LNI 192.168.202.154@tcp109 [8/256/0/180] [14648.629058] LNet: Added LNI 192.168.202.154@tcp110 [8/256/0/180] [14650.184418] LNet: Added LNI 192.168.202.154@tcp111 [8/256/0/180] [14651.458928] LNet: Added LNI 192.168.202.154@tcp112 [8/256/0/180] [14652.562452] LNet: Added LNI 192.168.202.154@tcp113 [8/256/0/180] [14653.867574] LNet: Added LNI 192.168.202.154@tcp114 [8/256/0/180] [14655.109878] LNet: Added LNI 192.168.202.154@tcp115 [8/256/0/180] [14656.450657] LNet: Added LNI 192.168.202.154@tcp116 [8/256/0/180] [14657.683224] LNet: Added LNI 192.168.202.154@tcp117 [8/256/0/180] [14658.725089] LNet: Added LNI 192.168.202.154@tcp118 [8/256/0/180] [14659.770854] LNet: Added LNI 192.168.202.154@tcp119 [8/256/0/180] [14660.981182] LNet: Added LNI 192.168.202.154@tcp120 [8/256/0/180] [14661.964484] LNet: Added LNI 192.168.202.154@tcp121 [8/256/0/180] [14663.350383] LNet: Added LNI 192.168.202.154@tcp122 [8/256/0/180] [14664.273888] LNet: Added LNI 192.168.202.154@tcp123 [8/256/0/180] [14665.365119] LNet: Added LNI 192.168.202.154@tcp124 [8/256/0/180] [14666.490189] LNet: Added LNI 192.168.202.154@tcp125 [8/256/0/180] [14667.233931] LNet: Added LNI 192.168.202.154@tcp126 [8/256/0/180] [14667.940909] LNet: Added LNI 192.168.202.154@tcp127 [8/256/0/180] [14668.925211] LNet: Added LNI 192.168.202.154@tcp128 [8/256/0/180] [14669.860469] LNet: Added LNI 192.168.202.154@tcp129 [8/256/0/180] [14670.749509] LNet: Added LNI 192.168.202.154@tcp130 [8/256/0/180] [14671.498452] LNet: Added LNI 192.168.202.154@tcp131 [8/256/0/180] [14672.340329] LNet: Added LNI 192.168.202.154@tcp132 [8/256/0/180] [14673.384486] LNet: Added LNI 192.168.202.154@tcp133 [8/256/0/180] [14674.442807] LNet: Added LNI 192.168.202.154@tcp134 [8/256/0/180] [14675.603291] LNet: Added LNI 192.168.202.154@tcp135 [8/256/0/180] [14683.495760] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14683.553726] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14684.681796] Lustre: 435216:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.154@tcp: can't add NID 192.168.202.154@tcp130: rc = -75 [14684.870684] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14684.907936] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14685.031848] Lustre: lustre-MDT0000: new disk, initializing [14685.193248] Lustre: 435267:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.154@tcp: can't add NID 192.168.202.154@tcp130: rc = -75 [14685.199992] Lustre: 435267:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [14685.240233] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14685.266729] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14689.318786] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [14700.937345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14701.067292] Lustre: 435233:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14701.124678] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14701.127790] Lustre: Skipped 1 previous similar message [14701.187662] Lustre: lustre-MDT0001: new disk, initializing [14701.220306] Lustre: 436013:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.154@tcp: can't add NID 192.168.202.154@tcp130: rc = -75 [14701.225710] Lustre: 436013:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [14701.319491] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14701.349233] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14701.362375] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14706.248435] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [14710.807995] Lustre: 436437:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.154@tcp: can't add NID 192.168.202.154@tcp130: rc = -75 [14710.813727] Lustre: 436437:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [14713.367731] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14717.508108] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14724.840865] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14725.344547] Lustre: lustre-OST0000: new disk, initializing [14725.350438] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14725.360868] Lustre: 436976:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14725.503107] Lustre: 436981:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.154@tcp: can't add NID 192.168.202.154@tcp130: rc = -75 [14725.509553] Lustre: 436981:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [14725.565256] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14732.664509] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [14735.399272] Lustre: 437415:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.154@tcp: can't add NID 192.168.202.154@tcp130: rc = -75 [14735.405326] Lustre: 437415:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [14735.470674] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14735.479370] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14735.664926] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14741.190681] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14751.204411] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14751.205435] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14751.209460] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14751.222212] Lustre: Skipped 1 previous similar message [14751.252954] Lustre: Skipped 1 previous similar message [14755.114419] Lustre: server umount lustre-OST0000 complete [14761.446158] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14761.460228] Lustre: Skipped 1 previous similar message [14761.460389] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14761.473040] Lustre: Skipped 1 previous similar message [14765.853690] Lustre: server umount lustre-MDT0000 complete [14769.791104] LustreError: 435224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781242843 with bad export cookie 4040124384803409339 [14769.792112] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14769.832356] LustreError: 435224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14770.123377] Lustre: server umount lustre-MDT0001 complete [14780.606403] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [14783.559445] Key type lgssc unregistered [14783.992661] LNet: 438758:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14784.002484] LNetError: 438758:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14784.013566] LNet: Removed LNI 192.168.202.154@tcp [14785.055295] LNet: Removed LNI 192.168.202.154@tcp100 [14785.060343] LNetError: 438758:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14785.080255] LNetError: 438758:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14785.094457] LNet: Removed LNI 192.168.202.154@tcp101 [14786.143309] LNet: Removed LNI 192.168.202.154@tcp102 [14786.150695] LNetError: 438758:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14786.167859] LNetError: 438758:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14786.181468] LNet: Removed LNI 192.168.202.154@tcp103 [14786.187523] LNet: Removed LNI 192.168.202.154@tcp104 [14786.192929] LNet: Removed LNI 192.168.202.154@tcp105 [14786.201768] LNet: Removed LNI 192.168.202.154@tcp106 [14786.205158] LNet: Removed LNI 192.168.202.154@tcp107 [14786.210936] LNet: Removed LNI 192.168.202.154@tcp108 [14786.218350] LNet: Removed LNI 192.168.202.154@tcp109 [14786.223839] LNet: Removed LNI 192.168.202.154@tcp110 [14786.235337] LNet: Removed LNI 192.168.202.154@tcp111 [14786.243073] LNet: Removed LNI 192.168.202.154@tcp112 [14786.245166] LNet: Removed LNI 192.168.202.154@tcp113 [14786.247960] LNet: Removed LNI 192.168.202.154@tcp114 [14786.250735] LNet: Removed LNI 192.168.202.154@tcp115 [14786.253309] LNet: Removed LNI 192.168.202.154@tcp116 [14786.258526] LNet: Removed LNI 192.168.202.154@tcp117 [14786.260990] LNet: Removed LNI 192.168.202.154@tcp118 [14786.272983] LNet: Removed LNI 192.168.202.154@tcp119 [14786.279116] LNet: Removed LNI 192.168.202.154@tcp120 [14786.284835] LNet: Removed LNI 192.168.202.154@tcp121 [14786.289961] LNet: Removed LNI 192.168.202.154@tcp122 [14786.298466] LNet: Removed LNI 192.168.202.154@tcp123 [14786.303256] LNet: Removed LNI 192.168.202.154@tcp124 [14786.313789] LNet: Removed LNI 192.168.202.154@tcp125 [14786.328576] LNet: Removed LNI 192.168.202.154@tcp126 [14786.336882] LNet: Removed LNI 192.168.202.154@tcp127 [14786.340314] LNet: Removed LNI 192.168.202.154@tcp128 [14786.343266] LNet: Removed LNI 192.168.202.154@tcp129 [14786.349842] LNet: Removed LNI 192.168.202.154@tcp130 [14786.356036] LNet: Removed LNI 192.168.202.154@tcp131 [14786.363359] LNet: Removed LNI 192.168.202.154@tcp132 [14786.365951] LNet: Removed LNI 192.168.202.154@tcp133 [14786.373525] LNet: Removed LNI 192.168.202.154@tcp134 [14786.411469] LNet: Removed LNI 192.168.202.154@tcp135 [14787.401436] Key type .llcrypt unregistered [14787.405532] Key type ._llcrypt unregistered [14811.273443] Key type ._llcrypt registered [14811.277032] Key type .llcrypt registered [14811.485159] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [14824.653881] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [14825.940782] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14825.958244] alg: No test for adler32 (adler32-zlib) [14827.101688] Lustre: Lustre: Build Version: 2.17.53_64_gab2e0dd [14827.380937] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [14829.081924] Key type lgssc registered [14830.286225] Lustre: Echo OBD driver; http://www.lustre.org/ [14866.332682] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 01:42:18 (1781242938) [14883.056185] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [14886.261446] Key type lgssc unregistered [14886.684108] LNet: 442784:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14886.691836] LNetError: 442784:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14886.711584] LNet: Removed LNI 192.168.202.154@tcp [14887.747967] Key type .llcrypt unregistered [14887.753570] Key type ._llcrypt unregistered [14903.203947] Key type ._llcrypt registered [14903.205905] Key type .llcrypt registered [14903.311681] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [14904.366837] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14904.460032] alg: No test for adler32 (adler32-zlib) [14905.476240] Lustre: Lustre: Build Version: 2.17.53_64_gab2e0dd [14905.757993] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [14907.455269] Key type lgssc registered [14908.518887] Lustre: Echo OBD driver; http://www.lustre.org/ [14917.975383] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14918.001820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14919.303969] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14919.357598] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14919.477644] Lustre: lustre-MDT0000: new disk, initializing [14919.575289] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14919.599543] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14923.607934] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [14934.246042] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14934.354594] Lustre: 443979:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14934.395532] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14934.398337] Lustre: Skipped 1 previous similar message [14934.458750] Lustre: lustre-MDT0001: new disk, initializing [14934.575879] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14934.632518] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14934.645618] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14939.582685] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [14946.121772] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14949.401513] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14955.765376] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14956.069654] Lustre: lustre-OST0000: new disk, initializing [14956.072963] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14956.078228] Lustre: 445723:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14956.173748] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14962.270344] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [14964.341370] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14964.368727] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14964.495018] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14970.679765] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14974.508975] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing lnet_if_list [14980.843725] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14980.880270] LNet: Added LNI 192.168.202.154@tcp10 [8/256/0/180] [14987.111135] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14987.140842] LNet: Added LNI 192.168.202.154@tcp11 [8/256/0/180] [14990.931597] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14990.975487] LNet: Added LNI 192.168.202.154@tcp12 [8/256/0/180] [14995.563200] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14995.638610] LNet: Added LNI 192.168.202.154@tcp13 [8/256/0/180] [15000.382061] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [15000.413210] LNet: Added LNI 192.168.202.154@tcp14 [8/256/0/180] [15004.492899] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [15004.542357] LNet: Added LNI 192.168.202.154@tcp15 [8/256/0/180] [15008.828298] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [15008.855749] LNet: Added LNI 192.168.202.154@tcp16 [8/256/0/180] [15013.449153] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [15013.470890] LNet: Added LNI 192.168.202.154@tcp17 [8/256/0/180] [15018.301764] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [15018.331578] LNet: Added LNI 192.168.202.154@tcp18 [8/256/0/180] [15022.752940] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [15022.771867] LNet: Added LNI 192.168.202.154@tcp19 [8/256/0/180] [15026.499633] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [15026.525861] LNet: Added LNI 192.168.202.154@tcp20 [8/256/0/180] [15030.550226] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [15030.579546] LNet: Added LNI 192.168.202.154@tcp21 [8/256/0/180] [15035.023937] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [15035.063836] LNet: Added LNI 192.168.202.154@tcp22 [8/256/0/180] [15039.967824] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [15039.993707] LNet: Added LNI 192.168.202.154@tcp23 [8/256/0/180] [15045.283736] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [15045.324606] LNet: Added LNI 192.168.202.154@tcp24 [8/256/0/180] [15050.542243] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [15050.578636] LNet: Added LNI 192.168.202.154@tcp25 [8/256/0/180] [15055.657588] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [15055.678079] LNet: Added LNI 192.168.202.154@tcp26 [8/256/0/180] [15060.810413] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [15060.848097] LNet: Added LNI 192.168.202.154@tcp27 [8/256/0/180] [15065.783481] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [15065.838359] LNet: Added LNI 192.168.202.154@tcp28 [8/256/0/180] [15071.386434] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [15071.410760] LNet: Added LNI 192.168.202.154@tcp29 [8/256/0/180] [15077.272353] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [15077.312714] LNet: Added LNI 192.168.202.154@tcp30 [8/256/0/180] [15083.768135] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [15083.793723] LNet: Added LNI 192.168.202.154@tcp31 [8/256/0/180] [15088.442939] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [15088.467785] LNet: Added LNI 192.168.202.154@tcp32 [8/256/0/180] [15094.204573] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [15094.260794] LNet: Added LNI 192.168.202.154@tcp33 [8/256/0/180] [15099.683132] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [15099.703213] LNet: Added LNI 192.168.202.154@tcp34 [8/256/0/180] [15105.340442] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [15105.372580] LNet: Added LNI 192.168.202.154@tcp35 [8/256/0/180] [15111.197647] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [15111.222288] LNet: Added LNI 192.168.202.154@tcp36 [8/256/0/180] [15117.590251] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [15117.608409] LNet: Added LNI 192.168.202.154@tcp37 [8/256/0/180] [15123.972989] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [15124.000857] LNet: Added LNI 192.168.202.154@tcp38 [8/256/0/180] [15129.642405] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [15129.663303] LNet: Added LNI 192.168.202.154@tcp39 [8/256/0/180] [15135.788993] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [15135.808911] LNet: Added LNI 192.168.202.154@tcp40 [8/256/0/180] [15141.059151] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [15141.081273] LNet: Added LNI 192.168.202.154@tcp41 [8/256/0/180] [15145.560279] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [15145.579730] LNet: Added LNI 192.168.202.154@tcp42 [8/256/0/180] [15151.284677] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [15151.321666] LNet: Added LNI 192.168.202.154@tcp43 [8/256/0/180] [15157.083383] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [15157.105911] LNet: Added LNI 192.168.202.154@tcp44 [8/256/0/180] [15163.337794] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [15163.367456] LNet: Added LNI 192.168.202.154@tcp45 [8/256/0/180] [15168.330166] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [15168.346364] LNet: Added LNI 192.168.202.154@tcp46 [8/256/0/180] [15173.415129] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [15173.437426] LNet: Added LNI 192.168.202.154@tcp47 [8/256/0/180] [15178.797075] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [15178.817746] LNet: Added LNI 192.168.202.154@tcp48 [8/256/0/180] [15184.140529] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [15184.157897] LNet: Added LNI 192.168.202.154@tcp49 [8/256/0/180] [15188.972548] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [15188.995461] LNet: Added LNI 192.168.202.154@tcp50 [8/256/0/180] [15200.225353] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15200.226753] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15200.231887] Lustre: Skipped 2 previous similar messages [15200.234433] Lustre: Skipped 1 previous similar message [15202.277841] Lustre: server umount lustre-MDT0000 complete [15204.889128] LustreError: 443968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781243278 with bad export cookie 18353730912378870213 [15204.904384] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15205.171695] Lustre: server umount lustre-MDT0001 complete [15217.880827] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [15221.665445] Lustre: 443252:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781243279/real 1781243279] req@00000000e1b8d66e x1867768638093312/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781243295 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15221.690475] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15224.048397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15225.503147] Lustre: 443253:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781243283/real 1781243283] req@00000000453ece74 x1867768638093440/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781243299 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15226.847169] Lustre: 443251:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781243284/real 1781243284] req@00000000a052ce32 x1867768638093568/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781243300 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15230.237991] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15230.947585] Lustre: 443250:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781243288/real 1781243288] req@000000005b58c460 x1867768638093824/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781243304 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15233.085661] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [15235.041384] Lustre: 443252:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781243293/real 1781243293] req@000000005b49efb1 x1867768638093952/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781243309 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15236.584461] LustreError: 469080:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15236.598085] LustreError: 469080:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [15239.093251] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15239.350609] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15240.361368] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15242.209390] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [15246.128840] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15248.917565] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [15265.252389] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15265.260768] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15265.266946] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15266.282963] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15266.286230] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15266.288446] Lustre: Skipped 1 previous similar message [15268.649905] Lustre: server umount lustre-OST0000 complete [15275.491477] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15275.503299] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15275.511470] Lustre: Skipped 1 previous similar message [15275.514283] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15277.749698] Lustre: server umount lustre-MDT0000 complete [15280.691257] LustreError: 445732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781243354 with bad export cookie 18353730912378872761 [15280.699183] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15280.702131] LustreError: 445732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [15281.045290] Lustre: server umount lustre-MDT0001 complete [15287.526093] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [15289.378791] Key type lgssc unregistered [15289.677234] LNet: 471616:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15289.685532] LNetError: 471616:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [15289.693731] LNet: Removed LNI 192.168.202.154@tcp [15289.701538] LNet: Removed LNI 192.168.202.154@tcp10 [15289.706935] LNet: Removed LNI 192.168.202.154@tcp11 [15289.710451] LNet: Removed LNI 192.168.202.154@tcp12 [15289.713580] LNet: Removed LNI 192.168.202.154@tcp13 [15289.715487] LNet: Removed LNI 192.168.202.154@tcp14 [15289.721958] LNet: Removed LNI 192.168.202.154@tcp15 [15289.723696] LNet: Removed LNI 192.168.202.154@tcp16 [15289.729665] LNet: Removed LNI 192.168.202.154@tcp17 [15289.731030] LNet: Removed LNI 192.168.202.154@tcp18 [15289.732038] LNet: Removed LNI 192.168.202.154@tcp19 [15289.734434] LNet: Removed LNI 192.168.202.154@tcp20 [15289.736360] LNet: Removed LNI 192.168.202.154@tcp21 [15289.738362] LNet: Removed LNI 192.168.202.154@tcp22 [15289.741613] LNet: Removed LNI 192.168.202.154@tcp23 [15289.746575] LNet: Removed LNI 192.168.202.154@tcp24 [15289.748740] LNet: Removed LNI 192.168.202.154@tcp25 [15289.750556] LNet: Removed LNI 192.168.202.154@tcp26 [15289.752670] LNet: Removed LNI 192.168.202.154@tcp27 [15289.754355] LNet: Removed LNI 192.168.202.154@tcp28 [15289.756089] LNet: Removed LNI 192.168.202.154@tcp29 [15289.757455] LNet: Removed LNI 192.168.202.154@tcp30 [15289.758864] LNet: Removed LNI 192.168.202.154@tcp31 [15289.761738] LNet: Removed LNI 192.168.202.154@tcp32 [15289.764982] LNet: Removed LNI 192.168.202.154@tcp33 [15289.768830] LNet: Removed LNI 192.168.202.154@tcp34 [15289.771946] LNet: Removed LNI 192.168.202.154@tcp35 [15289.776905] LNet: Removed LNI 192.168.202.154@tcp36 [15289.779694] LNet: Removed LNI 192.168.202.154@tcp37 [15289.782783] LNet: Removed LNI 192.168.202.154@tcp38 [15289.786385] LNet: Removed LNI 192.168.202.154@tcp39 [15289.788795] LNet: Removed LNI 192.168.202.154@tcp40 [15289.791830] LNet: Removed LNI 192.168.202.154@tcp41 [15289.794547] LNet: Removed LNI 192.168.202.154@tcp42 [15289.796225] LNet: Removed LNI 192.168.202.154@tcp43 [15289.798356] LNet: Removed LNI 192.168.202.154@tcp44 [15289.800848] LNet: Removed LNI 192.168.202.154@tcp45 [15289.803384] LNet: Removed LNI 192.168.202.154@tcp46 [15289.806298] LNet: Removed LNI 192.168.202.154@tcp47 [15289.808902] LNet: Removed LNI 192.168.202.154@tcp48 [15289.811818] LNet: Removed LNI 192.168.202.154@tcp49 [15289.819255] LNet: Removed LNI 192.168.202.154@tcp50 [15290.377601] Key type .llcrypt unregistered [15290.379742] Key type ._llcrypt unregistered [15305.973180] Key type ._llcrypt registered [15305.974309] Key type .llcrypt registered [15306.086559] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [15318.368474] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [15319.508283] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15319.698396] alg: No test for adler32 (adler32-zlib) [15320.686310] Lustre: Lustre: Build Version: 2.17.53_64_gab2e0dd [15320.881727] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [15322.524453] Key type lgssc registered [15323.478863] Lustre: Echo OBD driver; http://www.lustre.org/ [15352.551233] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15352.568111] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15353.806584] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15353.831441] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15353.897865] Lustre: lustre-MDT0000: new disk, initializing [15353.963661] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15353.973404] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15357.302847] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [15365.275625] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15365.360245] Lustre: 474758:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15365.410359] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15365.414149] Lustre: Skipped 1 previous similar message [15365.456193] Lustre: lustre-MDT0001: new disk, initializing [15365.533239] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15365.556602] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15365.570571] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15369.115375] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [15375.644849] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15378.765321] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [15384.828101] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15384.936346] Lustre: *** cfs_fail_loc=90f, val=0*** [15384.983597] Lustre: lustre-OST0000: new disk, initializing [15384.985896] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15384.990316] Lustre: 476547:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15385.069597] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15389.866224] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [15395.412607] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15395.419474] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15395.529954] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15395.985440] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [15400.932214] Lustre: server umount lustre-OST0000 complete [15406.049481] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15406.060516] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15406.064958] Lustre: Skipped 1 previous similar message [15411.172921] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15411.183872] Lustre: Skipped 2 previous similar messages [15413.281143] Lustre: server umount lustre-MDT0000 complete [15416.160746] LustreError: 474751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781243490 with bad export cookie 12382168209362256882 [15416.167612] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15416.171513] LustreError: 474751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [15426.653802] Lustre: server umount lustre-MDT0001 complete [15435.591307] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [15437.957293] Key type lgssc unregistered [15438.297161] LNet: 478322:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15438.302432] LNetError: 478322:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [15439.342464] LNet: Removed LNI 192.168.202.154@tcp [15439.904276] Key type .llcrypt unregistered [15439.907079] Key type ._llcrypt unregistered [15453.544261] Key type ._llcrypt registered [15453.549177] Key type .llcrypt registered [15453.627768] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [15454.390681] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15454.496263] alg: No test for adler32 (adler32-zlib) [15455.437558] Lustre: Lustre: Build Version: 2.17.53_64_gab2e0dd [15455.569346] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [15457.167187] Key type lgssc registered [15458.197488] Lustre: Echo OBD driver; http://www.lustre.org/ [15464.428602] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15464.444909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15465.823631] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15468.877012] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [15475.170085] LustreError: 479555:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15475.188091] LustreError: 479555:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [15475.495436] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15475.759857] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15479.364088] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [15483.413324] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15486.012178] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [15491.558772] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15491.883285] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15497.165694] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [15501.823346] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [15520.108222] loop0: detected capacity change from 0 to 204800000 [15520.224511] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15520.254760] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15520.266985] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15524.896321] loop0: detected capacity change from 0 to 204800000 [15525.056517] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15525.073715] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15525.084886] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15529.176553] loop0: detected capacity change from 0 to 204800000 [15533.802151] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15533.945774] Lustre: fs15246-OST0000: new disk, initializing [15533.949202] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [15533.955096] Lustre: 482830:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15534.000170] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15539.676698] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [15549.048476] loop1: detected capacity change from 0 to 204800000 [15554.124552] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15554.261160] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [15554.328188] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [15554.421473] Lustre: fs15246-MDT0000: new disk, initializing [15554.650037] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15554.690088] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15554.806672] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [15555.873338] Lustre: 483938:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@00000000698baa29 x1867769214706944/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1781243645 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [15558.435593] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [15560.942425] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [15560.952877] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [15561.012829] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15575.529048] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15575.560483] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15580.648134] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15580.760534] Lustre: server umount fs15246-MDT0000 complete [15585.767203] LustreError: 479548:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15587.887980] Lustre: server umount fs15246-OST0000 complete [15596.010797] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15596.015399] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15596.023082] Lustre: Skipped 1 previous similar message [15599.076733] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15599.081256] Lustre: Skipped 2 previous similar messages [15604.199769] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15609.311148] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15609.421160] Lustre: server umount lustre-OST0000 complete [15613.922912] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15613.928223] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15613.934262] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15613.935975] Lustre: Skipped 3 previous similar messages [15618.259115] Lustre: server umount lustre-MDT0000 complete [15619.553595] LustreError: 479555:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15619.570806] LustreError: 479555:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [15621.171344] LustreError: 479536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781243695 with bad export cookie 16685888952157582261 [15621.175159] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15621.178968] LustreError: 479536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [15621.433968] Lustre: server umount lustre-MDT0001 complete [15628.936964] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [15630.553210] Key type lgssc unregistered [15630.740283] LNet: 486998:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15630.743130] LNetError: 486998:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [15630.756880] LNet: Removed LNI 192.168.202.154@tcp [15631.285908] Key type .llcrypt unregistered [15631.287696] Key type ._llcrypt unregistered [15651.607401] Key type ._llcrypt registered [15651.610744] Key type .llcrypt registered [15651.833049] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [15653.551497] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15653.882688] alg: No test for adler32 (adler32-zlib) [15655.000582] Lustre: Lustre: Build Version: 2.17.53_64_gab2e0dd [15655.141768] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [15656.807184] Key type lgssc registered [15658.673800] Lustre: Echo OBD driver; http://www.lustre.org/ [15728.207717] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [15740.583128] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [15778.259574] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15778.281988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15779.623970] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15779.698706] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15779.824791] Lustre: lustre-MDT0000: new disk, initializing [15779.986507] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15780.022605] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15784.860160] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [15795.842404] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15795.994656] Lustre: 491420:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15796.114682] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15796.121908] Lustre: Skipped 1 previous similar message [15796.298861] Lustre: lustre-MDT0001: new disk, initializing [15796.423330] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15796.475151] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15796.494415] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15802.279782] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [15812.456547] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15817.607257] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [15825.869368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15826.226599] Lustre: lustre-OST0000: new disk, initializing [15826.230314] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15826.238228] Lustre: 493163:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15826.377524] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15834.643807] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [15835.789143] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15835.815388] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15835.971307] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15843.570398] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [15848.961093] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15849.209010] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15854.360243] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15854.500594] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15856.609841] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15856.612651] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15856.613369] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15856.619658] LustreError: Skipped 1 previous similar message [15856.648613] Lustre: Skipped 1 previous similar message [15861.747733] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15861.761554] Lustre: Skipped 2 previous similar messages [15862.725209] Lustre: server umount lustre-OST0000 complete [15866.850453] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15866.867178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15866.876118] Lustre: Skipped 1 previous similar message [15871.973101] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15877.090837] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15877.103234] Lustre: Skipped 2 previous similar messages [15881.183328] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15881.448702] Lustre: server umount lustre-MDT0000 complete [15884.887750] LustreError: 491411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781243958 with bad export cookie 15105733099809716906 [15884.888584] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15884.894693] LustreError: 491411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15885.195474] Lustre: server umount lustre-MDT0001 complete [15911.537771] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [15922.034462] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [15958.694409] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 15692 sec ========= 02:00:30 (1781244030) [15961.461148] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 02:00:32 (1781244032) === [15966.783469] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 02:00:38 (1781244038) === [16015.359605] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [16018.919261] Key type lgssc unregistered [16019.336618] LNet: 499628:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16019.340559] LNetError: 499628:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [16019.348580] LNet: Removed LNI 192.168.202.154@tcp [16020.278227] Key type .llcrypt unregistered [16020.279942] Key type ._llcrypt unregistered