[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 883421321 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.002427] x2apic enabled [ 0.003016] Switched APIC routing to physical x2apic. [ 0.004017] kvm-guest: setup PV IPIs [ 0.005794] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.006000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.006031] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.007012] pid_max: default: 32768 minimum: 301 [ 0.008135] LSM: Security Framework initializing [ 0.009056] Yama: becoming mindful. [ 0.010041] SELinux: Initializing. [ 0.011080] *** VALIDATE selinux *** [ 0.021088] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026198] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027150] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028148] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029122] *** VALIDATE tmpfs *** [ 0.030506] *** VALIDATE proc *** [ 0.032208] *** VALIDATE cgroup *** [ 0.033013] *** VALIDATE cgroup2 *** [ 0.035277] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036155] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037013] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038028] Spectre V2 : User space: Vulnerable [ 0.039009] Speculative Store Bypass: Vulnerable [ 0.042092] debug: unmapping init [mem 0xffffffffb8a59000-0xffffffffb8a60fff] [ 0.044273] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045694] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046029] ... version: 2 [ 0.047019] ... bit width: 48 [ 0.048014] ... generic registers: 4 [ 0.049026] ... value mask: 0000ffffffffffff [ 0.050022] ... max period: 00007fffffffffff [ 0.051022] ... fixed-purpose events: 3 [ 0.052018] ... event mask: 000000070000000f [ 0.053406] rcu: Hierarchical SRCU implementation. [ 0.056101] smp: Bringing up secondary CPUs ... [ 0.057778] x86: Booting SMP configuration: [ 0.058056] .... node #0, CPUs: #1 #2 #3 [ 0.068481] smp: Brought up 1 node, 4 CPUs [ 0.070046] smpboot: Max logical packages: 1 [ 0.071021] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.098059] node 0 deferred pages initialised in 25ms [ 0.102469] devtmpfs: initialized [ 0.103319] x86/mm: Memory block size: 128MB [ 0.106220] gcov: version magic: 0x41383552 [ 0.108148] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.109091] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.110382] pinctrl core: initialized pinctrl subsystem [ 0.111208] [ 0.111845] ************************************************************* [ 0.112014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.113013] ** ** [ 0.114206] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.115017] ** ** [ 0.116013] ** This means that this kernel is built to expose internal ** [ 0.117021] ** IOMMU data structures, which may compromise security on ** [ 0.118013] ** your system. ** [ 0.119016] ** ** [ 0.120152] ** If you see this message and you are not debugging the ** [ 0.121012] ** kernel, report this immediately to your vendor! ** [ 0.122012] ** ** [ 0.123013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.124017] ************************************************************* [ 0.125847] NET: Registered protocol family 16 [ 0.126607] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.127065] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.128082] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.129565] cpuidle: using governor menu [ 0.131926] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.134743] PCI: Using configuration type 1 for base access [ 0.137275] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.144113] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.145017] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.149030] cryptd: max_cpu_qlen set to 1000 [ 0.151363] ACPI: Added _OSI(Module Device) [ 0.152010] ACPI: Added _OSI(Processor Device) [ 0.153010] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.154009] ACPI: Added _OSI(Processor Aggregator Device) [ 0.159368] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.163222] ACPI: Interpreter enabled [ 0.164108] ACPI: PM: (supports S0 S3 S4 S5) [ 0.165013] ACPI: Using IOAPIC for interrupt routing [ 0.166087] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.167331] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.177569] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.178045] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.179033] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.180123] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.184155] acpiphp: Slot [2] registered [ 0.186078] acpiphp: Slot [5] registered [ 0.187137] acpiphp: Slot [6] registered [ 0.188120] acpiphp: Slot [7] registered [ 0.189150] acpiphp: Slot [8] registered [ 0.190265] acpiphp: Slot [9] registered [ 0.191288] acpiphp: Slot [10] registered [ 0.192136] acpiphp: Slot [3] registered [ 0.193257] acpiphp: Slot [4] registered [ 0.194200] acpiphp: Slot [11] registered [ 0.195820] acpiphp: Slot [12] registered [ 0.196391] acpiphp: Slot [13] registered [ 0.197314] acpiphp: Slot [14] registered [ 0.198345] acpiphp: Slot [15] registered [ 0.199153] acpiphp: Slot [16] registered [ 0.200138] acpiphp: Slot [17] registered [ 0.201206] acpiphp: Slot [18] registered [ 0.202180] acpiphp: Slot [19] registered [ 0.203180] acpiphp: Slot [20] registered [ 0.204323] acpiphp: Slot [21] registered [ 0.205351] acpiphp: Slot [22] registered [ 0.206197] acpiphp: Slot [23] registered [ 0.207222] acpiphp: Slot [24] registered [ 0.208255] acpiphp: Slot [25] registered [ 0.209193] acpiphp: Slot [26] registered [ 0.210242] acpiphp: Slot [27] registered [ 0.211131] acpiphp: Slot [28] registered [ 0.212186] acpiphp: Slot [29] registered [ 0.213171] acpiphp: Slot [30] registered [ 0.214206] acpiphp: Slot [31] registered [ 0.215187] PCI host bridge to bus 0000:00 [ 0.216034] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.217121] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.218028] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.219026] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.220023] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.221036] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.222168] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.224501] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.226190] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.234024] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.237668] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.238043] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.239098] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.240021] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.241601] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.249683] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.252037] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.255955] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.260011] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.272015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.278018] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.288569] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.324024] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.339027] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.361018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.446803] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.507024] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.518020] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.544017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.556725] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.574023] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.582020] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.618016] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.631619] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.645018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.656019] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.680016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.691865] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.700014] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.706021] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.729023] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.971009] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.985024] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.998027] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 1.009000] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 1.009000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 1.009000] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 1.009000] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 1.009000] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 1.009000] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 1.013069] iommu: Default domain type: Passthrough [ 1.044463] SCSI subsystem initialized [ 1.045000] ACPI: bus type USB registered [ 1.045201] usbcore: registered new interface driver usbfs [ 1.047070] usbcore: registered new interface driver hub [ 1.048000] usbcore: registered new device driver usb [ 1.049288] pps_core: LinuxPPS API ver. 1 registered [ 1.051020] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 1.055078] PTP clock support registered [ 1.057254] EDAC MC: Ver: 3.0.0 [ 1.058159] PCI: Using ACPI for IRQ routing [ 1.060058] NetLabel: Initializing [ 1.061007] NetLabel: domain hash size = 128 [ 1.063010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 1.065140] NetLabel: unlabeled traffic allowed by default [ 1.068147] vgaarb: loaded [ 1.069813] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.072009] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.079721] clocksource: Switched to clocksource kvm-clock [ 1.450505] VFS: Disk quotas dquot_6.6.0 [ 1.454550] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.582620] *** VALIDATE ramfs *** [ 1.585336] *** VALIDATE hugetlbfs *** [ 1.587187] pnp: PnP ACPI init [ 1.589594] pnp: PnP ACPI: found 6 devices [ 1.611405] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.614955] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.617183] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.619339] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.621730] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.624462] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.627244] NET: Registered protocol family 2 [ 1.629594] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.634568] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.639099] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.644486] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.647711] TCP: Hash tables configured (established 65536 bind 65536) [ 1.650340] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.653514] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.656396] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.659407] NET: Registered protocol family 1 [ 1.661693] RPC: Registered named UNIX socket transport module. [ 1.664111] RPC: Registered udp transport module. [ 1.665955] RPC: Registered tcp transport module. [ 1.667772] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.670505] NET: Registered protocol family 44 [ 1.672134] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.674961] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.677558] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.680159] PCI: CLS 0 bytes, default 64 [ 1.682409] Unpacking initramfs... [ 3.487243] debug: unmapping init [mem 0xffff949f3cc54000-0xffff949f3ffbffff] [ 3.491398] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.493540] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.496582] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 4.117098] Initialise system trusted keyrings [ 4.118431] Key type blacklist registered [ 4.119854] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 4.128236] zbud: loaded [ 4.130787] *** VALIDATE nfs *** [ 4.131783] *** VALIDATE nfs4 *** [ 4.133063] pstore: using deflate compression [ 4.136295] Platform Keyring initialized [ 4.248330] NET: Registered protocol family 38 [ 4.249850] Key type asymmetric registered [ 4.251571] Asymmetric key parser 'x509' registered [ 4.253502] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.256240] io scheduler mq-deadline registered [ 4.257667] io scheduler kyber registered [ 4.259344] io scheduler bfq registered [ 4.260856] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.263586] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.266312] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.268576] ACPI: Power Button [PWRF] [ 4.274128] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.280813] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.294205] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.301586] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.315779] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.344697] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.387281] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.396643] Non-volatile memory driver v1.3 [ 4.398224] Linux agpgart interface v0.103 [ 4.432548] virtio_blk virtio1: [vda] 146648 512-byte logical blocks (75.1 MB/71.6 MiB) [ 4.435679] vda: detected capacity change from 0 to 75083776 [ 4.455788] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.459136] vdb: detected capacity change from 0 to 1073741824 [ 4.514051] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.516936] vdc: detected capacity change from 0 to 2621440000 [ 4.591416] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.595860] vdd: detected capacity change from 0 to 2621440000 [ 4.619433] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.622758] vde: detected capacity change from 0 to 4294967296 [ 4.638368] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.641583] vdf: detected capacity change from 0 to 4294967296 [ 4.648415] libphy: Fixed MDIO Bus: probed [ 4.665942] usbcore: registered new interface driver usbserial_generic [ 4.667880] usbserial: USB Serial support registered for generic [ 4.669607] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.673691] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.675381] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.677516] mousedev: PS/2 mouse device common for all mice [ 4.680521] rtc_cmos 00:05: RTC can wake from S4 [ 4.683429] rtc_cmos 00:05: registered as rtc0 [ 4.685408] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.689025] intel_pstate: CPU model not supported [ 4.690624] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.695046] hid: raw HID events driver (C) Jiri Kosina [ 4.697776] usbcore: registered new interface driver usbhid [ 4.699547] usbhid: USB HID core driver [ 4.701484] drop_monitor: Initializing network drop monitor service [ 4.703625] Initializing XFRM netlink socket [ 4.705643] NET: Registered protocol family 10 [ 4.708391] Segment Routing with IPv6 [ 4.709523] NET: Registered protocol family 17 [ 4.710728] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.713645] mpls_gso: MPLS GSO support [ 4.726657] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.731719] RAS: Correctable Errors collector initialized. [ 4.737293] AVX version of gcm_enc/dec engaged. [ 4.738837] AES CTR mode by8 optimization enabled [ 4.834589] sched_clock: Marking stable (4834524535, 0)->(6783702816, -1949178281) [ 4.838976] registered taskstats version 1 [ 4.841803] Loading compiled-in X.509 certificates [ 4.844021] zswap: loaded using pool lzo/zbud [ 4.877856] Key type big_key registered [ 4.893908] Key type encrypted registered [ 4.897529] ima: No TPM chip found, activating TPM-bypass! [ 4.902418] ima: Allocated hash algorithm: sha1 [ 4.905786] ima: No architecture policies found [ 4.912375] evm: Initialising EVM extended attributes: [ 4.914660] evm: security.selinux [ 4.916828] evm: security.ima [ 4.918821] evm: security.capability [ 4.920376] evm: HMAC attrs: 0x1 [ 4.922973] rtc_cmos 00:05: setting system clock to 2026-09-06 16:30:34 UTC (1788712234) [ 4.931735] debug: unmapping init [mem 0xffffffffb9a03000-0xffffffffb9bfffff] [ 4.934965] debug: unmapping init [mem 0xffffffffb8782000-0xffffffffb8a58fff] [ 4.943186] Write protecting the kernel read-only data: 28672k [ 4.948704] debug: unmapping init [mem 0xffffffffb6e03000-0xffffffffb6ffffff] [ 4.951630] debug: unmapping init [mem 0xffffffffb7714000-0xffffffffb77fffff] [ 4.992415] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.000808] systemd[1]: Detected virtualization kvm. [ 5.002947] systemd[1]: Detected architecture x86-64. [ 5.005091] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.033963] systemd[1]: No hostname configured. [ 5.035780] systemd[1]: Set hostname to . [ 5.038110] random: systemd: uninitialized urandom read (16 bytes read) [ 5.040695] systemd[1]: Initializing machine ID from random generator. [ 5.401372] random: systemd: uninitialized urandom read (16 bytes read) [ 5.403755] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 5.410896] random: systemd: uninitialized urandom read (16 bytes read) [ 5.412960] systemd[1]: Reached target Paths. [ OK ] Reached target Paths. [ 5.418471] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Swap. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... Starting Setup Virtual Console... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local File Systems. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Timers. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 7.001734] device-mapper: uevent: version 1.0.3 [ 7.003633] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ 8.936975] virtio_net virtio0 ens2: renamed from eth0 [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 9.314838] random: fast init done [ 9.341917] scsi host0: ata_piix [ 9.398418] scsi host1: ata_piix [ 9.405335] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 9.408990] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 16.427549] random: crng init done [ 16.429362] random: 7 urandom warning(s) missed due to ratelimiting [ 18.245066] dracut-initqueue[582]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 20.371311] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 23.844919] printk: systemd: 26 output lines suppressed due to ratelimiting [ 24.779777] SELinux: Disabled at runtime. [ 24.886771] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 24.897450] systemd[1]: Detected virtualization kvm. [ 24.899882] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 26.884803] systemd[1]: initrd-switch-root.service: Succeeded. [ 26.897606] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 26.917100] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 26.926839] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 26.937802] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 26.956447] systemd[1]: Starting Journal Service... Starting Journal Service... [ 26.967918] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on udev Control Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. Starting Remount Root and Kernel File Systems... Mounting Kernel Debug File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-getty.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Initrd Root File System. Mounting POSIX Message Queue File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ 27.351085] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. Starting udev Coldplug all Devices... Starting Apply Kernel Variables... [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Huge Pages File System... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 28.433360] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 29.687877] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 29.871507] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 30.483356] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 30.754953] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit)[ 35.376757] Key type dns_resolver registered [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit)[ 36.569355] NFS: Registering the id_resolver key type [ 36.573563] Key type id_resolver registered [ 36.582427] Key type id_legacy registered [ ***] A start job is running for Configur…only root support (10s / no limit) [ **] A start job is running for Configur…only root support (10s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg221-server login: [ 58.474339] hrtimer: interrupt took 2980455 ns [ 96.460938] libcfs: loading out-of-tree module taints kernel. [ 96.494684] Key type ._llcrypt registered [ 96.496760] Key type .llcrypt registered [ 96.571777] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_hostid [ 113.032535] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 114.476703] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 114.489379] alg: No test for adler32 (adler32-zlib) [ 115.766584] Lustre: Lustre: Build Version: 2.17.57_103_g794a134 [ 116.553337] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 118.407228] Key type lgssc registered [ 119.937900] Lustre: Echo OBD driver; http://www.lustre.org/ [ 136.027841] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 171.736849] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 184.733509] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 184.755855] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 186.040976] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 186.071486] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 186.180316] Lustre: lustre-MDT0000: new disk, initializing [ 186.344166] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 186.357397] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 190.271166] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 203.449926] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 203.548629] Lustre: 6510:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 203.568925] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 203.573371] Lustre: Skipped 1 previous similar message [ 203.657653] Lustre: lustre-MDT0001: new disk, initializing [ 203.726397] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 203.749796] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 203.757512] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 208.109397] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 212.357404] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 220.162950] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 220.485863] Lustre: lustre-OST0000: new disk, initializing [ 220.490480] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 220.497574] Lustre: 8445:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 220.548719] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 222.051961] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 222.056730] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 222.169880] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 226.787741] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 239.933744] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 240.113893] Lustre: lustre-OST0001: new disk, initializing [ 240.116879] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 240.121750] Lustre: 9521:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 240.171228] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 245.821849] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 247.373356] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 247.383722] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 247.411427] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 256.497480] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 267.562500] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 272.081182] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing check_logdir /tmp/testlogs/ [ 275.987409] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing yml_node [ 279.185325] Lustre: DEBUG MARKER: Client: 2.17.57.103 [ 281.123561] Lustre: DEBUG MARKER: MDS: 2.17.57.103 [ 283.001917] Lustre: DEBUG MARKER: OSS: 2.17.57.103 [ 284.265260] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sun Sep 6 12:35:13 EDT 2026 [ 299.735722] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 301.065392] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 316.384541] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 316.391953] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 316.408732] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 318.949588] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 318.955375] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 318.969523] Lustre: Skipped 2 previous similar messages [ 318.981806] Lustre: Skipped 3 previous similar messages [ 321.848853] Lustre: server umount lustre-MDT0000 complete [ 325.506617] LustreError: 7923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788712555 with bad export cookie 7674153456703778015 [ 325.508052] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 325.515924] LustreError: 7923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 329.189745] LustreError: 6517:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 329.190347] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 329.208539] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 329.213030] LustreError: 6517:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 329.241202] Lustre: Skipped 1 previous similar message [ 330.208105] LustreError: 8540:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 331.856994] Lustre: server umount lustre-MDT0001 complete [ 335.600900] Lustre: server umount lustre-OST0000 complete [ 339.640454] Lustre: server umount lustre-OST0001 complete [ 345.905089] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_hostid [ 354.452829] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 386.617828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 386.809658] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 386.827681] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 386.898425] Lustre: lustre-MDT0000: new disk, initializing [ 386.948899] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 386.971390] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 390.727205] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 400.440984] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 400.534724] Lustre: 15070:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 400.582815] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 400.587718] Lustre: Skipped 1 previous similar message [ 400.667746] Lustre: lustre-MDT0001: new disk, initializing [ 400.744214] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 400.786463] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 400.796380] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 404.319593] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 411.746237] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 415.989933] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 421.647265] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 421.936365] Lustre: lustre-OST0000: new disk, initializing [ 421.941782] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 421.945271] Lustre: 17039:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 422.035686] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 423.175651] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 423.191203] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 423.291996] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 426.073052] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 432.238432] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 435.257922] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 435.411177] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 438.275825] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 438.455667] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 443.877525] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 443.877835] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 443.895232] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 443.900821] Lustre: Skipped 3 previous similar messages [ 445.827960] Lustre: server umount lustre-OST0000 complete [ 448.994919] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 448.999074] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 449.010146] Lustre: Skipped 2 previous similar messages [ 449.012389] Lustre: Skipped 1 previous similar message [ 454.268424] Lustre: server umount lustre-MDT0000 complete [ 456.285892] LustreError: 17038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788712685 with bad export cookie 7674153456703779548 [ 456.286243] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 456.292709] LustreError: 17038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 456.479521] Lustre: server umount lustre-MDT0001 complete [ 458.657924] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 12:38:08 (1788712688) [ 460.200806] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 461.396806] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 12:38:11 (1788712691) [ 465.514875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 465.775627] LustreError: 19444:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 465.783805] LustreError: 19444:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 465.818894] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 467.668494] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 470.993397] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 471.008472] LustreError: 19445:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 472.663356] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 475.428565] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 477.473096] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 480.213355] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 482.616621] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 485.144825] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 501.220921] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 501.222626] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 501.225991] Lustre: Skipped 1 previous similar message [ 501.228987] Lustre: Skipped 3 previous similar messages [ 501.232763] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 501.236674] LustreError: Skipped 1 previous similar message [ 504.596909] Lustre: server umount lustre-OST0000 complete [ 520.159131] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 520.221531] Lustre: server umount lustre-MDT0000 complete [ 520.672356] LustreError: 19444:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 520.679707] LustreError: 19444:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 521.512165] LustreError: 20178:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788712751 with bad export cookie 7674153456703780430 [ 521.514354] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 521.517171] LustreError: 20178:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 521.631412] Lustre: server umount lustre-MDT0001 complete [ 525.129611] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 526.159121] Key type lgssc unregistered [ 526.284360] LNet: 22997:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 526.286774] LNetError: 22997:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 526.297282] LNet: Removed LNI 192.168.202.121@tcp [ 526.657131] Key type .llcrypt unregistered [ 526.658234] Key type ._llcrypt unregistered [ 535.210365] Key type ._llcrypt registered [ 535.212017] Key type .llcrypt registered [ 535.248808] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 535.720447] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 535.728736] alg: No test for adler32 (adler32-zlib) [ 536.614248] Lustre: Lustre: Build Version: 2.17.57_103_g794a134 [ 536.710516] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 538.287156] Key type lgssc registered [ 538.665585] Lustre: Echo OBD driver; http://www.lustre.org/ [ 541.309188] loop: module loaded [ 541.327272] loop0: detected capacity change from 0 to 204800000 [ 541.401187] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 541.405601] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 541.522649] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 543.448805] loop0: detected capacity change from 0 to 204800000 [ 543.510290] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 543.514282] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 543.519427] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 545.261456] loop0: detected capacity change from 0 to 204800000 [ 547.283693] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 547.289120] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 548.369965] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 548.383673] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 548.417539] Lustre: lustre-MDT0000: new disk, initializing [ 548.438322] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 548.444473] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 549.818433] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 553.545887] loop1: detected capacity change from 0 to 204800000 [ 555.412202] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 555.508413] Lustre: lustre-OST07c6: new disk, initializing [ 555.509905] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 555.511270] Lustre: Skipped 1 previous similar message [ 555.512849] Lustre: 26168:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 555.534074] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 557.518623] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 559.614097] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 559.617722] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 559.627516] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 564.885114] Lustre: 25184:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 564.887934] Lustre: 25184:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 564.890373] Lustre: 25184:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 564.892882] Lustre: 25184:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 564.895279] Lustre: 25184:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 564.898503] Lustre: 25184:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 577.033054] Lustre: 25184:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 577.035649] Lustre: 25184:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 577.037556] Lustre: 25184:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 577.040172] Lustre: 25184:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 577.043893] Lustre: 25184:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 577.047458] Lustre: 25184:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 580.539970] Lustre: 25184:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 580.542809] Lustre: 25184:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 580.545578] Lustre: 25184:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 580.547628] Lustre: 25184:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 580.550278] Lustre: 25184:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 580.553110] Lustre: 25184:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 580.555358] Lustre: 25184:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 580.557485] Lustre: 25184:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 580.560710] Lustre: 25184:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 580.563028] Lustre: 25184:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 580.565781] Lustre: 25184:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 580.569101] Lustre: 25184:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 583.621323] Lustre: 25155:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 583.625350] Lustre: 25155:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 583.628584] Lustre: 25155:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 583.631803] Lustre: 25155:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 583.635017] Lustre: 25155:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 583.637310] Lustre: 25155:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 583.640875] Lustre: 25155:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 583.644659] Lustre: 25155:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 583.648119] Lustre: 25155:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 583.651155] Lustre: 25155:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 583.654423] Lustre: 25155:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 583.656817] Lustre: 25155:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 590.165538] Lustre: 26687:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 590.168201] Lustre: 26687:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages [ 590.170241] Lustre: 26687:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 590.172127] Lustre: 26687:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 590.174236] Lustre: 26687:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 590.176207] Lustre: 26687:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 590.178227] Lustre: 26687:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 590.181537] Lustre: 26687:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 590.184024] Lustre: 26687:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 590.186533] Lustre: 26687:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 590.189543] Lustre: 26687:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 590.192607] Lustre: 26687:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 599.757357] Lustre: 26687:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 599.761908] Lustre: 26687:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 599.764901] Lustre: 26687:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 599.768081] Lustre: 26687:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 599.771435] Lustre: 26687:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 599.775024] Lustre: 26687:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 599.778706] Lustre: 26687:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 599.782206] Lustre: 26687:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 599.785180] Lustre: 26687:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 599.787069] Lustre: 26687:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 599.788973] Lustre: 26687:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 599.791526] Lustre: 26687:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 641.505893] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 641.505965] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 641.510926] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 646.624428] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 646.742304] Lustre: server umount lustre-OST07c6 complete [ 650.819128] Lustre: server umount lustre-MDT0000 complete [ 657.107820] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 658.314890] Key type lgssc unregistered [ 658.455359] LNet: 28747:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 658.457910] LNetError: 28747:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 658.466691] LNet: Removed LNI 192.168.202.121@tcp [ 658.809115] Key type .llcrypt unregistered [ 658.810758] Key type ._llcrypt unregistered [ 666.995100] Key type ._llcrypt registered [ 666.996762] Key type .llcrypt registered [ 667.041165] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 667.378316] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 667.386225] alg: No test for adler32 (adler32-zlib) [ 668.252201] Lustre: Lustre: Build Version: 2.17.57_103_g794a134 [ 668.360814] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 669.943176] Key type lgssc registered [ 670.406518] Lustre: Echo OBD driver; http://www.lustre.org/ [ 673.413970] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 673.423178] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 674.572207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 676.173431] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 679.440714] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 679.581270] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 681.130803] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 684.208827] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 686.501813] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 689.208175] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 689.357523] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 691.754545] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 694.383514] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 699.622433] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 704.993492] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 704.993601] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 704.995225] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 705.000475] Lustre: Skipped 1 previous similar message [ 710.114088] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 710.115713] Lustre: Skipped 2 previous similar messages [ 710.541261] Lustre: server umount lustre-OST0000 complete [ 715.232646] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 715.237549] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 715.239346] Lustre: Skipped 1 previous similar message [ 718.097417] Lustre: server umount lustre-MDT0000 complete [ 719.431280] LustreError: 29972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788712949 with bad export cookie 1281792442915988026 [ 719.434607] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 719.435497] LustreError: 29972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 719.555257] Lustre: server umount lustre-MDT0001 complete [ 723.400148] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 724.541860] Key type lgssc unregistered [ 724.685573] LNet: 33724:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 724.688944] LNetError: 33724:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 724.701533] LNet: Removed LNI 192.168.202.121@tcp [ 725.037812] Key type .llcrypt unregistered [ 725.039280] Key type ._llcrypt unregistered [ 732.569955] Key type ._llcrypt registered [ 732.571307] Key type .llcrypt registered [ 732.625348] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 733.006198] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 733.039414] alg: No test for adler32 (adler32-zlib) [ 733.897586] Lustre: Lustre: Build Version: 2.17.57_103_g794a134 [ 733.991277] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 735.575144] Key type lgssc registered [ 735.954511] Lustre: Echo OBD driver; http://www.lustre.org/ [ 738.703756] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 738.710026] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 739.871395] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 741.404263] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 744.504134] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 744.647776] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 746.042519] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 748.647804] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 750.500812] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 752.790343] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 752.914799] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 754.916426] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 757.453363] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 765.224791] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 775.649208] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 775.651134] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 775.652015] LustreError: Skipped 1 previous similar message [ 775.655962] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 775.658759] Lustre: Skipped 1 previous similar message [ 778.002145] Lustre: server umount lustre-OST0000 complete [ 780.768721] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 780.768762] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 780.775399] Lustre: Skipped 1 previous similar message [ 780.777801] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 780.780427] Lustre: Skipped 1 previous similar message [ 785.365506] Lustre: server umount lustre-MDT0000 complete [ 785.888029] LustreError: 35703:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 785.892810] LustreError: 35703:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 786.579107] LustreError: 34948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788713016 with bad export cookie 16482240105096287898 [ 786.582297] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 786.582463] LustreError: 34948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 786.689505] Lustre: server umount lustre-MDT0001 complete [ 790.146643] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 791.139820] Key type lgssc unregistered [ 791.266348] LNet: 38514:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 791.269610] LNetError: 38514:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 791.275282] LNet: Removed LNI 192.168.202.121@tcp [ 791.572755] Key type .llcrypt unregistered [ 791.573959] Key type ._llcrypt unregistered [ 800.055186] Key type ._llcrypt registered [ 800.056239] Key type .llcrypt registered [ 800.093279] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 800.361658] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 800.370267] alg: No test for adler32 (adler32-zlib) [ 801.221050] Lustre: Lustre: Build Version: 2.17.57_103_g794a134 [ 801.312569] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 802.887135] Key type lgssc registered [ 803.244799] Lustre: Echo OBD driver; http://www.lustre.org/ [ 805.854247] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 805.862088] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 807.015501] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 808.353237] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 811.140044] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 811.262177] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 812.618818] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 815.059033] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 816.834543] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 818.988505] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 819.096823] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 820.980826] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 823.516965] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 833.445639] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 837.087904] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 837.090408] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 837.095435] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 838.623983] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 838.625257] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 838.629546] Lustre: Skipped 1 previous similar message [ 842.579706] Lustre: server umount lustre-MDT0000 complete [ 843.745817] LustreError: 39904:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 843.751710] LustreError: 39904:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 843.789115] LustreError: 39883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788713073 with bad export cookie 7886467719325185284 [ 843.792451] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 843.794932] LustreError: 39883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 844.376769] LustreError: 39900:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 844.384325] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.21@tcp (stopping) [ 844.387979] Lustre: Skipped 3 previous similar messages [ 848.863863] LustreError: 42391:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 848.864101] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 848.875121] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 853.983881] LustreError: 39899:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 853.984156] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 853.988484] LustreError: 39899:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 853.990105] Lustre: Skipped 1 previous similar message [ 858.079085] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 858.154937] Lustre: server umount lustre-MDT0001 complete [ 884.193335] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 884.268159] Lustre: server umount lustre-OST0000 complete [ 889.562806] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 890.534576] Key type lgssc unregistered [ 890.670401] LNet: 43782:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 890.672959] LNetError: 43782:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 890.683397] LNet: Removed LNI 192.168.202.121@tcp [ 890.995502] Key type .llcrypt unregistered [ 890.997092] Key type ._llcrypt unregistered [ 899.398136] Key type ._llcrypt registered [ 899.398993] Key type .llcrypt registered [ 899.435763] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 899.708748] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 899.727320] alg: No test for adler32 (adler32-zlib) [ 900.565533] Lustre: Lustre: Build Version: 2.17.57_103_g794a134 [ 900.641593] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 902.215143] Key type lgssc registered [ 902.562412] Lustre: Echo OBD driver; http://www.lustre.org/ [ 905.026549] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 905.031653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 906.166926] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 907.408345] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 910.073397] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 910.189097] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 911.447870] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 913.597737] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 915.217160] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 917.188182] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 917.288551] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 918.990259] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 921.387383] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 931.623345] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 935.904328] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 935.906808] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 935.909929] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 936.929790] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 936.930508] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 936.936185] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 940.109783] Lustre: server umount lustre-OST0000 complete [ 956.383925] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 956.387176] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 956.392297] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 956.395547] Lustre: Skipped 2 previous similar messages [ 962.016348] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 962.018580] Lustre: Skipped 3 previous similar messages [ 967.135141] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 967.136207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 967.140993] Lustre: Skipped 1 previous similar message [ 967.198247] Lustre: server umount lustre-MDT0000 complete [ 968.504048] LustreError: 47014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788713198 with bad export cookie 4478335599611525535 [ 968.507175] LustreError: 47014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 968.507583] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 974.710389] Lustre: server umount lustre-MDT0001 complete [ 978.237226] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 979.258598] Key type lgssc unregistered [ 979.386380] LNet: 48893:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 979.390670] LNetError: 48893:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 979.398327] LNet: Removed LNI 192.168.202.121@tcp [ 979.731665] Key type .llcrypt unregistered [ 979.732849] Key type ._llcrypt unregistered [ 988.569725] Key type ._llcrypt registered [ 988.570953] Key type .llcrypt registered [ 988.608407] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 988.907037] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 988.914255] alg: No test for adler32 (adler32-zlib) [ 989.781190] Lustre: Lustre: Build Version: 2.17.57_103_g794a134 [ 989.879270] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 991.471189] Key type lgssc registered [ 991.851279] Lustre: Echo OBD driver; http://www.lustre.org/ [ 994.550792] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 994.556151] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 995.697781] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 997.042875] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 999.907950] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1000.033840] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1001.548937] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1004.139549] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1005.977777] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1008.241454] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1008.347980] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1010.365674] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1012.849386] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1021.670602] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1025.610437] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1026.707572] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1037.426695] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-0612h47m47s [ 1037.962920] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1038.949118] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.21@tcp (stopping) [ 1039.960378] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.21@tcp (stopping) [ 1040.863887] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1040.866838] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1041.888469] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1041.890166] Lustre: Skipped 4 previous similar messages [ 1044.884782] Lustre: server umount lustre-MDT0000 complete [ 1045.987400] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1046.103649] LustreError: 50292:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1046.113030] LustreError: 50292:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1048.032849] LustreError: 50298:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1048.040659] LustreError: 50298:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1048.220628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1048.267042] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1048.361145] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1048.386604] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1049.777155] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1050.914266] Lustre: DEBUG MARKER: Wait for df (36595) ... [ 1051.429877] Lustre: DEBUG MARKER: done [ 1053.668794] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1053.669566] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1053.672552] Lustre: Skipped 2 previous similar messages [ 1053.675614] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1053.677460] Lustre: Skipped 1 previous similar message [ 1054.485722] Lustre: server umount lustre-OST0000 complete [ 1054.697392] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1057.879722] Lustre: server umount lustre-MDT0000 complete [ 1059.174864] LustreError: 52137:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788713288 with bad export cookie 17839122872180695791 [ 1059.180320] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1059.181095] LustreError: 52137:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1065.337595] Lustre: server umount lustre-MDT0001 complete [ 1069.004663] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 1070.078895] Key type lgssc unregistered [ 1070.206417] LNet: 55228:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1070.211980] LNetError: 55228:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1070.222339] LNet: Removed LNI 192.168.202.121@tcp [ 1070.540438] Key type .llcrypt unregistered [ 1070.542416] Key type ._llcrypt unregistered [ 1084.746448] Key type ._llcrypt registered [ 1084.748225] Key type .llcrypt registered [ 1084.787620] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 1085.108561] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1085.155967] alg: No test for adler32 (adler32-zlib) [ 1086.005170] Lustre: Lustre: Build Version: 2.17.57_103_g794a134 [ 1086.100236] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 1087.687141] Key type lgssc registered [ 1088.039941] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1090.588742] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1090.595297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1091.694595] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1091.701737] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1091.760269] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1093.153856] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1096.036923] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1096.070685] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1096.083568] Lustre: 57408:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1096.185383] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1097.558864] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1099.954072] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1101.888234] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1104.177510] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1104.247419] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1104.327832] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1105.174956] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1106.365383] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1108.840920] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1112.176387] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-0612h49m02s [ 1112.696878] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1114.992248] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1115.840290] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1115.853808] LustreError: 58433:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1136.234245] Lustre: lustre-MDT0000: Client 4054c918-65d0-4220-8fc5-7b8e30d23065 (at 192.168.202.21@tcp) reconnecting [ 1155.895114] LustreError: 58433:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1156.612112] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1157.464180] Lustre: DEBUG MARKER: done [ 1161.697504] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1161.698120] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1161.700518] LustreError: Skipped 1 previous similar message [ 1161.704725] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1161.707833] Lustre: Skipped 1 previous similar message [ 1165.201413] Lustre: server umount lustre-OST0000 complete [ 1166.816514] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1166.817210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1166.820891] Lustre: Skipped 2 previous similar messages [ 1167.841096] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1167.843345] Lustre: Skipped 2 previous similar messages [ 1168.546215] Lustre: server umount lustre-MDT0000 complete [ 1169.819218] LustreError: 58150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788713399 with bad export cookie 7376236613010561151 [ 1169.822851] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1169.823765] LustreError: 58150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1175.991841] Lustre: server umount lustre-MDT0001 complete [ 1179.731352] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 1180.907234] Key type lgssc unregistered [ 1181.045434] LNet: 61729:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1181.049162] LNetError: 61729:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1181.056657] LNet: Removed LNI 192.168.202.121@tcp [ 1181.423876] Key type .llcrypt unregistered [ 1181.424876] Key type ._llcrypt unregistered [ 1197.695653] Key type ._llcrypt registered [ 1197.696554] Key type .llcrypt registered [ 1197.730913] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 1198.059251] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1198.110034] alg: No test for adler32 (adler32-zlib) [ 1198.960967] Lustre: Lustre: Build Version: 2.17.57_103_g794a134 [ 1199.046080] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 1200.631153] Key type lgssc registered [ 1200.986891] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1202.110592] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1202.116129] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1203.199770] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1203.207854] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1203.263080] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1203.586349] Lustre: Failing over lustre-MDT0000 [ 1203.678928] Lustre: server umount lustre-MDT0000 complete [ 1205.829417] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 12:50:36 (1788713436) [ 1208.136539] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1208.292278] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1209.655913] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1212.477528] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1212.510283] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1212.521470] Lustre: 64496:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1212.613404] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1214.019063] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1216.509114] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1218.360435] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1220.583051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1220.653313] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1220.725466] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1222.670844] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1225.084879] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1228.305416] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1233.309995] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1238.497100] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1238.497642] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1238.506369] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1238.508380] Lustre: Skipped 2 previous similar messages [ 1241.111115] Lustre: server umount lustre-MDT0000 complete [ 1242.333077] LustreError: 64490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788713471 with bad export cookie 9203841711199845042 [ 1242.336666] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1242.337041] LustreError: 64490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1242.439617] Lustre: server umount lustre-MDT0001 complete [ 1243.530205] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1245.043757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1247.874289] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1251.743791] LustreError: 63419:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000001adb7852 x1875601971510400/t0(0) o250->MGC192.168.202.121@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1251.892430] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1251.927931] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1253.252678] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1256.035170] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1256.134853] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1256.139046] Lustre: Skipped 2 previous similar messages [ 1257.504140] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1259.238225] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1259.892151] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1260.255168] Lustre: 63420:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788713473/real 1788713473] req@00000000cb4ec97e x1875601971510016/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788713489 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1261.730518] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1264.415174] Lustre: 63421:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788713477/real 1788713477] req@0000000031cf4c9e x1875601971510144/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788713493 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1264.609908] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1264.610985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1264.614682] Lustre: Skipped 1 previous similar message [ 1265.631176] Lustre: 63422:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788713478/real 1788713478] req@00000000288fc49d x1875601971510272/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788713494 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1267.679267] Lustre: 63423:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788713481/real 1788713481] req@0000000044b41114 x1875601971510528/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788713497 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1269.535627] Lustre: server umount lustre-MDT0000 complete [ 1269.729510] LustreError: 68266:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1269.735529] LustreError: 68266:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 1270.801624] LustreError: 64490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788713500 with bad export cookie 9203841711199847310 [ 1270.806365] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1270.806593] LustreError: 64490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1270.926472] Lustre: server umount lustre-MDT0001 complete [ 1273.131865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1276.253033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1280.992053] LustreError: 63419:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000d71ab911 x1875601971529728/t0(0) o250->MGC192.168.202.121@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1281.132400] LustreError: 66346:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1281.133496] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1281.144222] Lustre: Skipped 1 previous similar message [ 1281.165779] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1281.167656] Lustre: Skipped 1 previous similar message [ 1281.186766] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1282.632834] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1285.491979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1286.989420] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1289.378159] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1290.719301] Lustre: 63423:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788713504/real 1788713504] req@00000000a0da7e91 x1875601971529344/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788713520 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1290.729849] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1290.733413] Lustre: Skipped 1 previous similar message [ 1291.174712] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1294.944897] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1294.947140] Lustre: Skipped 2 previous similar messages [ 1294.947520] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1294.952396] Lustre: Skipped 1 previous similar message [ 1299.025618] Lustre: server umount lustre-MDT0000 complete [ 1300.230336] LustreError: 65233:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788713529 with bad export cookie 9203841711199848689 [ 1300.233527] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1300.233753] LustreError: 65233:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1300.341112] Lustre: server umount lustre-MDT0001 complete [ 1302.364897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1303.406253] Lustre: DEBUG MARKER: files compared the same [ 1314.404134] Lustre: server umount lustre-OST0000 complete [ 1320.023437] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 1320.986634] Key type lgssc unregistered [ 1321.127308] LNet: 74214:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1321.129425] LNetError: 74214:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1321.137254] LNet: Removed LNI 192.168.202.121@tcp [ 1321.449916] Key type .llcrypt unregistered [ 1321.451115] Key type ._llcrypt unregistered [ 1330.272832] Key type ._llcrypt registered [ 1330.273897] Key type .llcrypt registered [ 1330.310460] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 1330.638855] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1330.648273] alg: No test for adler32 (adler32-zlib) [ 1331.501589] Lustre: Lustre: Build Version: 2.17.57_103_g794a134 [ 1331.590906] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 1333.167136] Key type lgssc registered [ 1333.518331] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1336.084497] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1336.092052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1337.237669] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1338.630320] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug +malloc all [ 1341.559206] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1341.666700] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1343.098631] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug +malloc all [ 1345.553467] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1347.387022] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1349.694100] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1349.801563] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1351.754517] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug +malloc all [ 1354.214445] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1362.082168] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1367.519979] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1367.520187] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1367.521986] LustreError: Skipped 1 previous similar message [ 1367.525049] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1367.528118] Lustre: Skipped 1 previous similar message [ 1369.098964] Lustre: server umount lustre-OST0000 complete [ 1372.438314] Lustre: server umount lustre-MDT0000 complete [ 1372.639993] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1372.640411] LustreError: 78103:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1372.643481] Lustre: Skipped 2 previous similar messages [ 1372.643568] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1372.647738] LustreError: 78103:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1373.758822] LustreError: 78114:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788713603 with bad export cookie 9379203560636266673 [ 1373.760819] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1373.762054] LustreError: 78114:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1373.875328] Lustre: server umount lustre-MDT0001 complete [ 1377.644590] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 1378.674676] Key type lgssc unregistered [ 1378.794362] LNet: 79164:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1378.796623] LNetError: 79164:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1378.805282] LNet: Removed LNI 192.168.202.121@tcp [ 1379.108559] Key type .llcrypt unregistered [ 1379.110352] Key type ._llcrypt unregistered [ 1388.410633] Key type ._llcrypt registered [ 1388.412434] Key type .llcrypt registered [ 1388.456156] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 1388.843435] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1388.867265] alg: No test for adler32 (adler32-zlib) [ 1389.739808] Lustre: Lustre: Build Version: 2.17.57_103_g794a134 [ 1389.841445] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 1391.423107] Key type lgssc registered [ 1391.843110] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1394.512079] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1394.520595] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1409.951299] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1415.071264] LustreError: 80526:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.121@tcp: failed processing log, type 4: rc = -110 [ 1441.695159] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1443.628255] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1446.056847] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1448.583491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1448.624106] Lustre: *** cfs_fail_loc=706, val=0*** [ 1448.749791] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1448.769273] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 1450.059335] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1452.973204] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1453.104326] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1454.463736] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1456.873428] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1458.689296] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1462.243110] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1462.243950] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1462.247520] Lustre: Skipped 1 previous similar message [ 1462.252203] Lustre: Skipped 1 previous similar message [ 1465.485821] Lustre: server umount lustre-OST0000 complete [ 1467.360724] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1467.367697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1467.369518] Lustre: Skipped 1 previous similar message [ 1472.480299] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1472.482657] Lustre: Skipped 1 previous similar message [ 1472.853683] Lustre: server umount lustre-MDT0000 complete [ 1474.095135] LustreError: 83387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788713703 with bad export cookie 11513735353368050526 [ 1474.098288] LustreError: 83387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1474.098368] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1474.211539] Lustre: server umount lustre-MDT0001 complete [ 1477.625704] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 1478.664784] Key type lgssc unregistered [ 1478.790371] LNet: 84136:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1478.793155] LNetError: 84136:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1478.805277] LNet: Removed LNI 192.168.202.121@tcp [ 1479.100746] Key type .llcrypt unregistered [ 1479.101980] Key type ._llcrypt unregistered [ 1488.228185] Key type ._llcrypt registered [ 1488.229276] Key type .llcrypt registered [ 1488.269894] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 1488.672959] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1488.722991] alg: No test for adler32 (adler32-zlib) [ 1489.574213] Lustre: Lustre: Build Version: 2.17.57_103_g794a134 [ 1489.659518] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 1491.239156] Key type lgssc registered [ 1491.588047] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1494.774522] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1494.787613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1497.224050] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1499.850119] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1500.057299] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1501.426640] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1504.492628] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1504.616740] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1506.652985] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1509.498227] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1509.503982] Lustre: MGS: Client 38be3b27-258d-426a-b3a7-0cb0c403b2d2 (at 0@lo) reconnecting [ 1509.514225] LustreError: 86226:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 1509.599882] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1509.634462] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 1510.941916] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1520.096198] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1520.098236] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1520.101713] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1521.122538] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1521.122935] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1521.123286] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1521.134076] Lustre: Skipped 1 previous similar message [ 1524.759057] Lustre: server umount lustre-OST0000 complete [ 1526.241297] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1526.241437] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1526.252175] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1526.257473] Lustre: Skipped 1 previous similar message [ 1531.360342] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1531.362591] Lustre: Skipped 1 previous similar message [ 1532.169830] Lustre: server umount lustre-MDT0000 complete [ 1533.529823] Lustre: server umount lustre-MDT0001 complete [ 1534.805773] Lustre: server umount MGS complete [ 1538.437541] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 12:56:08 (1788713768) [ 1548.291414] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_hostid [ 1551.569418] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 1563.515524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1564.911778] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1568.768326] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1568.871628] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 1568.891801] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1568.929749] Lustre: lustre-MDT0001: new disk, initializing [ 1568.959185] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1568.968160] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1568.970729] Lustre: 93369:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 1568.974531] LustreError: 93369:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 1570.359577] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1574.492088] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1574.588864] Lustre: lustre-OST0000: new disk, initializing [ 1574.590423] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1574.592704] Lustre: 94134:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1574.615218] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1576.498606] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1580.217819] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1580.575727] Lustre: 94789:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000b629d17d x1875602276648192/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788713826 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1582.422690] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1582.426352] Lustre: MGS: Client e03b82ab-42ba-4cdb-9e05-812c5ef3834c (at 0@lo) reconnecting [ 1582.441629] Lustre: 95180:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1582.448329] LustreError: 93354:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 1582.452213] LustreError: 93354:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 1582.456471] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1582.494465] Lustre: lustre-MDT0000: new disk, initializing [ 1582.545294] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1583.583994] Lustre: 95205:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000082521f1f x1875602276653696/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1788713829 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1583.647652] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1583.961711] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1588.642362] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1588.646598] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1588.711805] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 1609.183906] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1609.186398] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1609.189566] Lustre: Skipped 1 previous similar message [ 1609.191932] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1613.982958] Lustre: server umount lustre-OST0000 complete [ 1617.376769] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1617.377093] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1617.379130] LustreError: Skipped 1 previous similar message [ 1617.383090] Lustre: Skipped 1 previous similar message [ 1617.384797] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1617.387088] Lustre: Skipped 4 previous similar messages [ 1621.569779] Lustre: server umount lustre-MDT0000 complete [ 1622.496134] LustreError: 93339:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1622.502695] LustreError: 93339:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 1637.343141] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1637.427684] Lustre: server umount lustre-MDT0001 complete [ 1638.650563] Lustre: server umount MGS complete [ 1642.390596] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 12:57:52 (1788713872) [ 1648.340865] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 1649.337874] Key type lgssc unregistered [ 1649.459378] LNet: 97702:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1649.461511] LNetError: 97702:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1649.471340] LNet: Removed LNI 192.168.202.121@tcp [ 1649.780664] Key type .llcrypt unregistered [ 1649.781699] Key type ._llcrypt unregistered [ 1656.626341] Key type ._llcrypt registered [ 1656.627635] Key type .llcrypt registered [ 1656.668773] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 1656.988556] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1656.995453] alg: No test for adler32 (adler32-zlib) [ 1657.845781] Lustre: Lustre: Build Version: 2.17.57_103_g794a134 [ 1657.940187] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 1659.519126] Key type lgssc registered [ 1659.922382] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1662.531785] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1662.541136] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1663.693699] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1665.024581] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1667.893690] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1668.007973] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1669.316268] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1671.711639] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1673.442247] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1675.779520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1675.906175] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1677.820587] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1680.209517] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1689.257991] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 1693.664733] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1693.668861] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1693.674409] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1694.688767] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1694.689891] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1694.690463] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1699.278897] Lustre: server umount lustre-OST0000 complete [ 1703.904086] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1703.906258] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1703.909968] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1703.911706] Lustre: Skipped 1 previous similar message [ 1706.653567] Lustre: server umount lustre-MDT0000 complete [ 1707.876153] LustreError: 98879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788713937 with bad export cookie 12992392367812227033 [ 1707.879196] LustreError: 98879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1707.879857] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1707.998815] Lustre: server umount lustre-MDT0001 complete [ 1711.815682] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 1712.996270] Key type lgssc unregistered [ 1713.143698] LNet: 102445:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1713.147735] LNetError: 102445:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1713.156469] LNet: Removed LNI 192.168.202.121@tcp [ 1713.478822] Key type .llcrypt unregistered [ 1713.479922] Key type ._llcrypt unregistered [ 1719.925170] Key type ._llcrypt registered [ 1719.926213] Key type .llcrypt registered [ 1719.966273] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 1720.314830] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1720.387366] alg: No test for adler32 (adler32-zlib) [ 1721.261885] Lustre: Lustre: Build Version: 2.17.57_103_g794a134 [ 1721.361540] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 1722.943187] Key type lgssc registered [ 1723.331209] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1725.407259] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1725.407365] LustreError: 103501:0:(tgt_mount.c:2570:server_fill_super()) cfs_race id 716 sleeping [ 1725.415233] Lustre: Skipped 1 previous similar message [ 1725.417659] LustreError: 103500:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 waking [ 1725.421797] LustreError: 103501:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 awake: rc=4986 [ 1725.423902] LustreError: 103501:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 1725.427592] LustreError: 103501:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1725.429456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1726.603455] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1729.150537] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1729.246673] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1730.565090] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1732.282353] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing lsmod [ 1733.273162] LustreError: 104704:0:(tgt_mount.c:2570:server_fill_super()) cfs_race id 716 sleeping [ 1733.273295] LustreError: 104705:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 waking [ 1733.278039] LustreError: 104704:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 awake: rc=4997 [ 1733.280042] LustreError: 104704:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 1733.283022] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1733.283753] LustreError: 104704:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1733.408047] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1734.623926] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1734.626168] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1734.629902] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1738.720796] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1738.721870] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1738.723853] Lustre: Skipped 1 previous similar message [ 1738.728489] Lustre: Skipped 1 previous similar message [ 1740.436699] Lustre: server umount lustre-MDT0000 complete [ 1741.633229] LustreError: 103515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788713971 with bad export cookie 10788103079902985584 [ 1741.635580] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1741.636382] LustreError: 103515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1742.815229] LustreError: 103968:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000043ea12c4 x1875602518850560/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1742.863776] Lustre: server umount lustre-MDT0001 complete [ 1754.323606] Lustre: server umount lustre-OST0000 complete [ 1757.135314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1757.284204] LustreError: 105769:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1757.289404] LustreError: 105769:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1757.307437] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1758.643396] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1761.513225] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1761.655577] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1763.129281] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1765.564614] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1767.390135] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1770.119775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1770.252765] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1772.164493] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1774.491638] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1781.548499] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 1786.850828] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1786.851402] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1786.851665] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1786.855769] Lustre: Skipped 1 previous similar message [ 1788.234097] Lustre: server umount lustre-OST0000 complete [ 1791.581273] Lustre: server umount lustre-MDT0000 complete [ 1791.970660] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1791.971593] LustreError: 105765:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1791.974937] Lustre: Skipped 1 previous similar message [ 1792.873347] LustreError: 105751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788714022 with bad export cookie 10788103079902986256 [ 1792.876609] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1792.878747] LustreError: 105751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1792.994713] Lustre: server umount lustre-MDT0001 complete [ 1796.545272] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 1797.517198] Key type lgssc unregistered [ 1797.637364] LNet: 109318:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1797.639470] LNetError: 109318:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1797.646330] LNet: Removed LNI 192.168.202.121@tcp [ 1797.954075] Key type .llcrypt unregistered [ 1797.955903] Key type ._llcrypt unregistered [ 1806.863090] Key type ._llcrypt registered [ 1806.864074] Key type .llcrypt registered [ 1806.906825] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 1807.415899] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1807.447089] alg: No test for adler32 (adler32-zlib) [ 1808.320977] Lustre: Lustre: Build Version: 2.17.57_103_g794a134 [ 1808.422719] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 1810.015087] Key type lgssc registered [ 1810.375650] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1812.995319] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1813.002358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1814.152063] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1815.413526] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1818.194761] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1818.314488] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1819.583505] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1821.997449] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1823.829649] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1826.186189] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1826.313636] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1828.240207] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1830.715891] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1838.628727] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 1843.187902] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 1844.126247] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 1848.807973] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1848.808120] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1848.812850] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1848.827043] Lustre: Skipped 1 previous similar message [ 1851.274613] Lustre: server umount lustre-OST0000 complete [ 1853.921138] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1853.921667] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1853.924652] Lustre: Skipped 1 previous similar message [ 1853.929359] Lustre: Skipped 2 previous similar messages [ 1858.666397] Lustre: server umount lustre-MDT0000 complete [ 1859.040233] LustreError: 110706:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1859.045686] LustreError: 110706:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1859.945082] LustreError: 110687:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788714089 with bad export cookie 15441466564681960281 [ 1859.947932] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1859.948041] LustreError: 110687:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1860.060619] Lustre: server umount lustre-MDT0001 complete [ 1863.942116] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 1864.958008] Key type lgssc unregistered [ 1865.100666] LNet: 114372:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1865.104570] LNetError: 114372:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1865.115391] LNet: Removed LNI 192.168.202.121@tcp [ 1865.454447] Key type .llcrypt unregistered [ 1865.456787] Key type ._llcrypt unregistered [ 1872.255915] Key type ._llcrypt registered [ 1872.256896] Key type .llcrypt registered [ 1872.293706] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 1872.591794] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1872.642480] alg: No test for adler32 (adler32-zlib) [ 1873.483342] Lustre: Lustre: Build Version: 2.17.57_103_g794a134 [ 1873.568735] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 1875.143102] Key type lgssc registered [ 1875.493122] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1878.158121] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1878.167229] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1879.318460] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1880.689326] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1883.459382] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1883.594137] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1885.008417] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1887.430335] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1889.333108] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1891.656171] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1891.794912] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1893.795770] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1896.241543] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1904.105824] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 1914.336970] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1914.339470] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1914.340779] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1914.343309] Lustre: Skipped 1 previous similar message [ 1915.473253] Lustre: server umount lustre-OST0000 complete [ 1919.457024] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1919.457123] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1919.458067] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1919.458072] Lustre: Skipped 1 previous similar message [ 1919.460394] LustreError: Skipped 1 previous similar message [ 1919.465232] Lustre: Skipped 1 previous similar message [ 1922.839887] Lustre: server umount lustre-MDT0000 complete [ 1924.087447] LustreError: 115546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788714153 with bad export cookie 15895456741252254185 [ 1924.090590] LustreError: 115546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1924.091327] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1924.218094] Lustre: server umount lustre-MDT0001 complete [ 1927.795477] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 1928.791626] Key type lgssc unregistered [ 1928.917356] LNet: 119118:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1928.920645] LNetError: 119118:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1928.927300] LNet: Removed LNI 192.168.202.121@tcp [ 1929.242020] Key type .llcrypt unregistered [ 1929.243077] Key type ._llcrypt unregistered [ 1938.078777] Key type ._llcrypt registered [ 1938.079715] Key type .llcrypt registered [ 1938.114048] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 1938.482476] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1938.505223] alg: No test for adler32 (adler32-zlib) [ 1939.348467] Lustre: Lustre: Build Version: 2.17.57_103_g794a134 [ 1939.427320] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 1940.999181] Key type lgssc registered [ 1941.331380] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1943.673304] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1943.678428] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1944.819280] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1946.092382] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1948.826728] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1948.949994] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1950.367445] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1952.893412] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1954.883977] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1957.472089] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1957.629394] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1959.890415] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1962.305478] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1969.957637] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 1975.249476] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 1976.530502] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 1976.532496] Lustre: Skipped 2 previous similar messages [ 1977.982457] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 1977.985269] Lustre: Skipped 2 previous similar messages [ 1983.992857] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 1984.003195] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 1986.908777] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.21@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 1986.914110] Lustre: Skipped 2 previous similar messages [ 1990.070433] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.21@tcp 192.168.0.[2,10]@tcp [ 1992.128875] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.21/24@tcp in log lustre-MDT0000 [ 1992.131745] Lustre: Skipped 2 previous similar messages [ 1996.023865] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.21/24@tcp [ 1996.026148] Lustre: Skipped 1 previous similar message [ 1999.839790] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1999.841630] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1999.844821] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2000.865502] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2000.866036] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2000.866263] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2001.492622] Lustre: server umount lustre-OST0000 complete [ 2005.471969] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2005.474968] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2005.480958] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2005.484141] Lustre: Skipped 1 previous similar message [ 2011.104780] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2011.106713] Lustre: Skipped 2 previous similar messages [ 2016.224220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2016.226372] Lustre: Skipped 1 previous similar message [ 2017.247159] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2017.322196] Lustre: server umount lustre-MDT0000 complete [ 2018.656404] LustreError: 120488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788714248 with bad export cookie 17335910457098978130 [ 2018.659491] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2018.661282] LustreError: 120488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2018.786368] Lustre: server umount lustre-MDT0001 complete [ 2022.475192] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 2023.491228] Key type lgssc unregistered [ 2023.614396] LNet: 125602:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2023.618557] LNetError: 125602:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2023.627331] LNet: Removed LNI 192.168.202.121@tcp [ 2023.970665] Key type .llcrypt unregistered [ 2023.971876] Key type ._llcrypt unregistered [ 2034.345473] Key type ._llcrypt registered [ 2034.346461] Key type .llcrypt registered [ 2034.389268] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 2034.792220] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2034.804322] alg: No test for adler32 (adler32-zlib) [ 2035.651950] Lustre: Lustre: Build Version: 2.17.57_103_g794a134 [ 2035.736937] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 2037.319185] Key type lgssc registered [ 2037.678486] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2040.453100] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2040.460745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2041.634122] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 2041.636265] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2041.643549] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.21/24@tcp [ 2043.064464] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2045.900908] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2046.005313] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2046.016475] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.21/24@tcp [ 2046.029299] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2047.443123] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2049.972815] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2051.823782] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2054.122306] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2054.245915] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2056.161990] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2058.571866] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2066.535427] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 2077.153528] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2077.154713] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2077.159693] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2077.162426] Lustre: Skipped 1 previous similar message [ 2082.273151] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2082.274865] Lustre: Skipped 1 previous similar message [ 2086.367114] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2086.416890] Lustre: server umount lustre-OST0000 complete [ 2092.000023] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2092.002446] LustreError: Skipped 1 previous similar message [ 2092.003940] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2092.008153] Lustre: Skipped 1 previous similar message [ 2092.010084] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2093.778688] Lustre: server umount lustre-MDT0000 complete [ 2095.041305] LustreError: 127114:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788714324 with bad export cookie 12962936846325463974 [ 2095.045372] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2095.046925] LustreError: 127114:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2095.172307] Lustre: server umount lustre-MDT0001 complete [ 2098.915072] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 2099.859287] Key type lgssc unregistered [ 2099.984424] LNet: 130742:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2099.986846] LNetError: 130742:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2099.994312] LNet: Removed LNI 192.168.202.121@tcp [ 2100.321704] Key type .llcrypt unregistered [ 2100.323653] Key type ._llcrypt unregistered [ 2108.950709] Key type ._llcrypt registered [ 2108.951882] Key type .llcrypt registered [ 2108.994639] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_hostid [ 2115.005713] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 2115.481898] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2115.499068] alg: No test for adler32 (adler32-zlib) [ 2116.353329] Lustre: Lustre: Build Version: 2.17.57_103_g794a134 [ 2116.449743] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 2118.031169] Key type lgssc registered [ 2118.408465] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2129.394581] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2129.400479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2130.484946] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2130.495341] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2130.535383] Lustre: lustre-MDT0000: new disk, initializing [ 2130.555961] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2130.561266] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2131.944439] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2136.066792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2136.110101] Lustre: 134114:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2136.132662] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2136.135436] Lustre: Skipped 1 previous similar message [ 2136.169993] Lustre: lustre-MDT0001: new disk, initializing [ 2136.202029] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2136.214146] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2136.218419] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2137.542512] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2141.256965] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2143.034684] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2145.288230] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2145.394979] Lustre: lustre-OST0000: new disk, initializing [ 2145.397215] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2145.400253] Lustre: 136084:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2145.431583] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2146.741873] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2146.744973] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2146.779823] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2147.306597] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2150.969793] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2152.358607] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2152.430065] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2153.770657] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2153.846186] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2157.026969] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2157.027403] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2157.030842] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2157.032640] LustreError: Skipped 1 previous similar message [ 2157.034756] Lustre: Skipped 1 previous similar message [ 2160.590327] Lustre: server umount lustre-OST0000 complete [ 2162.144736] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2162.144921] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2162.146084] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2162.152013] Lustre: Skipped 2 previous similar messages [ 2167.264048] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2167.266973] Lustre: Skipped 2 previous similar messages [ 2167.973120] Lustre: server umount lustre-MDT0000 complete [ 2169.271197] LustreError: 134151:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788714398 with bad export cookie 18387231201744062026 [ 2169.275723] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2169.396346] Lustre: server umount lustre-MDT0001 complete [ 2172.457060] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2172.633959] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2174.005883] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2177.097451] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2177.207359] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2178.548830] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2180.906109] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2182.692727] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2184.932861] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2186.946213] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2189.345030] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2190.756471] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2200.031139] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 9 sec [ 2201.481564] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2201.570468] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2204.890328] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2204.958793] Lustre: lustre-OST0001: new disk, initializing [ 2204.961179] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2204.964367] Lustre: 141731:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2204.990940] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2204.993393] Lustre: Skipped 1 previous similar message [ 2206.959546] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2210.179011] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2212.349846] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2212.353466] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2212.382325] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2213.310125] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 2214.652859] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2214.731690] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2216.550974] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9bd0d15e6800.ost_server_uuid 50 [ 2217.078658] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9bd0d15e6800.ost_server_uuid in FULL state after 0 sec [ 2222.563986] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2222.564259] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2222.566538] LustreError: Skipped 1 previous similar message [ 2222.571275] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2222.574220] Lustre: Skipped 1 previous similar message [ 2225.018857] Lustre: server umount lustre-OST0001 complete [ 2227.681481] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2227.683057] LustreError: 143053:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2227.686509] Lustre: Skipped 1 previous similar message [ 2227.687127] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2227.690873] LustreError: 143053:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2227.693131] Lustre: Skipped 1 previous similar message [ 2232.332768] Lustre: server umount lustre-OST0000 complete [ 2233.823934] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2233.826354] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2233.829620] Lustre: Skipped 1 previous similar message [ 2237.920237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2237.924588] Lustre: Skipped 2 previous similar messages [ 2239.702685] Lustre: server umount lustre-MDT0000 complete [ 2240.918288] LustreError: 138278:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788714470 with bad export cookie 18387231201744062901 [ 2240.921363] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2240.921516] LustreError: 138278:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 2241.034257] Lustre: server umount lustre-MDT0001 complete [ 2247.644749] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 2248.701954] Key type lgssc unregistered [ 2248.827392] LNet: 144748:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2248.830759] LNetError: 144748:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2248.839290] LNet: Removed LNI 192.168.202.121@tcp [ 2249.154134] Key type .llcrypt unregistered [ 2249.155255] Key type ._llcrypt unregistered [ 2262.250294] Key type ._llcrypt registered [ 2262.251287] Key type .llcrypt registered [ 2262.288145] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_hostid [ 2268.575561] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 2269.011378] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2269.019278] alg: No test for adler32 (adler32-zlib) [ 2269.873292] Lustre: Lustre: Build Version: 2.17.57_103_g794a134 [ 2269.964416] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 2271.543158] Key type lgssc registered [ 2271.944112] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2283.015560] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2283.020498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2284.099432] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2284.110284] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2284.150234] Lustre: lustre-MDT0000: new disk, initializing [ 2284.170997] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2284.176191] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2285.551611] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2289.618064] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2289.662219] Lustre: 148872:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2289.687111] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2289.688909] Lustre: Skipped 1 previous similar message [ 2289.725602] Lustre: lustre-MDT0001: new disk, initializing [ 2289.758542] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2289.772298] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2289.775287] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2291.134852] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2294.895651] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2296.868191] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2299.150757] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2299.253667] Lustre: lustre-OST0000: new disk, initializing [ 2299.255457] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2299.258498] Lustre: 150839:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2299.290089] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2301.185598] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2304.888665] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2306.551414] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2306.554618] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2306.587042] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2312.061541] Lustre: Failing over lustre-OST0000 [ 2312.132858] Lustre: server umount lustre-OST0000 complete [ 2325.033806] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2325.145562] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2325.151623] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2325.153047] Lustre: Skipped 1 previous similar message [ 2325.158725] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2325.159166] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2327.145129] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2328.413777] Lustre: Failing over lustre-MDT0000 [ 2328.503154] Lustre: server umount lustre-MDT0000 complete [ 2330.592845] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2330.594179] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2330.598860] Lustre: Skipped 2 previous similar messages [ 2332.639197] Lustre: 147099:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788714546/real 1788714546] req@0000000097cec351 x1875603094531968/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788714562 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2335.712171] LustreError: 150432:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2335.717012] LustreError: 150432:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 2336.856843] LustreError: 151557:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2337.759104] Lustre: 147096:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788714551/real 1788714551] req@00000000b20f90c8 x1875603094534016/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788714567 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2340.832345] LustreError: 148882:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2340.840578] LustreError: 148882:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2341.512626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2341.563656] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2341.569328] Lustre: 148874:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.121@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 2341.650543] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 2341.667667] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2341.702364] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2341.976627] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2343.097934] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2346.982138] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2346.988535] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2347.013427] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2352.099432] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2352.099527] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2352.103866] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2352.107496] LustreError: Skipped 1 previous similar message [ 2352.111597] Lustre: Skipped 1 previous similar message [ 2354.129581] Lustre: server umount lustre-OST0000 complete [ 2357.216906] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2357.217175] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2357.218106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2357.227140] Lustre: Skipped 2 previous similar messages [ 2361.554422] Lustre: server umount lustre-MDT0000 complete [ 2362.335968] LustreError: 148882:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2362.341048] LustreError: 148882:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2362.840042] LustreError: 148864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788714592 with bad export cookie 10560799387670875088 [ 2362.842560] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2362.844129] LustreError: 148864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2362.964456] Lustre: server umount lustre-MDT0001 complete [ 2366.857238] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 2367.901399] Key type lgssc unregistered [ 2368.032479] LNet: 154482:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2368.035804] LNetError: 154482:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2368.045340] LNet: Removed LNI 192.168.202.121@tcp [ 2368.377631] Key type .llcrypt unregistered [ 2368.379079] Key type ._llcrypt unregistered [ 2388.619939] Key type ._llcrypt registered [ 2388.620951] Key type .llcrypt registered [ 2388.663674] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 2389.197370] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2389.221483] alg: No test for adler32 (adler32-zlib) [ 2390.080654] Lustre: Lustre: Build Version: 2.17.57_103_g794a134 [ 2390.184793] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 2391.775140] Key type lgssc registered [ 2392.141161] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2394.695879] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2394.701489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2395.838877] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2397.281162] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2401.083295] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2403.115304] LustreError: 156026:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2403.120477] LustreError: 156026:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2403.125290] LustreError: 156026:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 0, failed: rc = -5 [ 2403.242779] Lustre: server umount lustre-MDT0000 complete [ 2408.260086] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_hostid [ 2411.551757] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 2423.280038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2423.374143] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2423.384147] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2423.422064] Lustre: lustre-MDT0000: new disk, initializing [ 2423.442905] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2423.450107] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2424.741822] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2428.904844] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2428.949965] Lustre: 159321:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2428.973731] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2428.976067] Lustre: Skipped 1 previous similar message [ 2429.015723] Lustre: lustre-MDT0001: new disk, initializing [ 2429.050514] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2429.062134] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2429.068715] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2430.419916] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2434.160894] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2435.985812] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2438.250345] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2438.350577] Lustre: lustre-OST0000: new disk, initializing [ 2438.352201] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2438.354868] Lustre: 161288:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2438.379632] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2439.486837] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2439.490286] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2439.536085] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2440.368523] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2444.058979] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2445.962427] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 2446.566923] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 13:11:16 (1788714676) [ 2511.327516] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2511.328138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2511.328141] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2511.340140] Lustre: Skipped 1 previous similar message [ 2515.889077] Lustre: server umount lustre-MDT0000 complete [ 2516.447580] LustreError: 159330:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2516.454258] LustreError: 159330:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2517.234716] LustreError: 163053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788714746 with bad export cookie 10016944246614932971 [ 2517.235028] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2517.239150] LustreError: 163053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2521.567410] LustreError: 160884:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2521.567455] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2521.578850] Lustre: Skipped 1 previous similar message [ 2521.580190] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2521.583438] Lustre: Skipped 2 previous similar messages [ 2522.591398] LustreError: 162007:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2522.591406] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2523.420694] Lustre: server umount lustre-MDT0001 complete [ 2530.760207] Lustre: server umount lustre-OST0000 complete [ 2533.650277] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_hostid [ 2536.961118] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 2548.545934] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2548.626752] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2548.636453] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2548.678309] Lustre: lustre-MDT0000: new disk, initializing [ 2548.699087] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2548.705400] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2550.066266] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2554.154234] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2554.197291] Lustre: 165913:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2554.221457] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2554.224181] Lustre: Skipped 1 previous similar message [ 2554.261965] Lustre: lustre-MDT0001: new disk, initializing [ 2554.303209] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2554.307836] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2555.645694] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2559.320738] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2561.183483] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2563.467152] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2563.564766] Lustre: lustre-OST0000: new disk, initializing [ 2563.566414] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2563.568757] Lustre: 167882:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2563.597480] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2563.599373] Lustre: Skipped 1 previous similar message [ 2564.095947] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2564.098990] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2564.126207] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2565.466757] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2569.083130] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2570.439433] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2570.505499] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2571.864445] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2571.935282] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2574.307649] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2574.308317] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2574.310528] LustreError: Skipped 1 previous similar message [ 2574.314736] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2574.316958] Lustre: Skipped 1 previous similar message [ 2578.730407] Lustre: server umount lustre-OST0000 complete [ 2584.545277] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2584.546277] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2584.550800] Lustre: Skipped 2 previous similar messages [ 2586.592643] Lustre: server umount lustre-MDT0000 complete [ 2587.862173] LustreError: 165905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788714817 with bad export cookie 10016944246615091787 [ 2587.864484] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2587.868826] LustreError: 165905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2588.002222] Lustre: server umount lustre-MDT0001 complete [ 2590.871080] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 13:13:41 (1788714821) [ 2596.524278] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_hostid [ 2599.825666] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 2611.783537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2611.869488] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2611.879432] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2611.918155] Lustre: lustre-MDT0000: new disk, initializing [ 2611.938098] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2611.944191] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2613.277983] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2617.414044] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2617.462746] Lustre: 172629:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2617.569883] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2618.952730] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2622.553273] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2624.318549] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2626.514162] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2626.611801] Lustre: 174603:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2626.854676] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2626.887757] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2628.526050] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2632.138468] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2637.285056] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2637.285091] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2637.289232] LustreError: Skipped 1 previous similar message [ 2637.293366] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2637.295472] Lustre: Skipped 2 previous similar messages [ 2640.149501] Lustre: server umount lustre-OST0000 complete [ 2647.577239] Lustre: server umount lustre-MDT0000 complete [ 2648.802205] LustreError: 175419:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788714878 with bad export cookie 10016944246615092655 [ 2648.804908] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2648.807852] LustreError: 175419:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2648.920886] Lustre: server umount lustre-MDT0001 complete [ 2651.608266] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 13:14:41 (1788714881) [ 2656.698350] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_hostid [ 2660.052212] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 2671.764218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2671.846671] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2671.856827] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2671.859551] Lustre: Skipped 3 previous similar messages [ 2671.898522] Lustre: lustre-MDT0000: new disk, initializing [ 2671.899764] Lustre: Skipped 2 previous similar messages [ 2671.918988] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2671.921060] Lustre: Skipped 2 previous similar messages [ 2671.925867] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2671.928067] Lustre: Skipped 2 previous similar messages [ 2673.250464] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2677.331309] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2677.383766] Lustre: 178916:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2677.494696] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2678.841683] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2682.579914] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2684.468411] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2686.657318] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2686.762952] Lustre: 180894:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2688.666392] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2692.344509] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2693.658217] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2703.842567] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2703.843636] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2703.845923] LustreError: Skipped 1 previous similar message [ 2703.847073] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2703.849859] Lustre: Skipped 4 previous similar messages [ 2703.852137] Lustre: Skipped 4 previous similar messages [ 2705.619514] Lustre: server umount lustre-OST0000 complete [ 2708.448053] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2721.247108] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2721.301991] Lustre: server umount lustre-MDT0000 complete [ 2722.560640] LustreError: 181712:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788714952 with bad export cookie 10016944246615093719 [ 2722.564751] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2722.564989] LustreError: 181712:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2722.691941] Lustre: server umount lustre-MDT0001 complete [ 2726.470449] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 2727.518904] Key type lgssc unregistered [ 2727.648356] LNet: 182749:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2727.650440] LNetError: 182749:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2727.659257] LNet: Removed LNI 192.168.202.121@tcp [ 2727.960215] Key type .llcrypt unregistered [ 2727.961316] Key type ._llcrypt unregistered [ 2737.032293] Key type ._llcrypt registered [ 2737.033949] Key type .llcrypt registered [ 2737.073590] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 2737.376305] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2737.382485] alg: No test for adler32 (adler32-zlib) [ 2738.231632] Lustre: Lustre: Build Version: 2.17.57_103_g794a134 [ 2738.321084] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 2739.903150] Key type lgssc registered [ 2740.240500] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2742.775809] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2742.781698] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2743.922354] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2745.336670] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2748.111681] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2748.249995] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2749.670072] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2752.304719] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2754.366825] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2756.872311] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2757.034604] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2759.006080] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2761.477993] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2770.343331] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2783.711935] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2783.714702] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2783.718596] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2785.761227] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2785.762981] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2785.767224] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2789.775658] Lustre: server umount lustre-OST0000 complete [ 2794.463945] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2794.466122] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2794.469526] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2794.471188] Lustre: Skipped 1 previous similar message [ 2797.214894] Lustre: server umount lustre-MDT0000 complete [ 2798.587209] LustreError: 184120:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788715028 with bad export cookie 1503806836685460670 [ 2798.589694] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2798.592395] LustreError: 184120:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2798.713318] Lustre: server umount lustre-MDT0001 complete [ 2802.573107] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 2803.720958] Key type lgssc unregistered [ 2803.851744] LNet: 187688:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2803.855645] LNetError: 187688:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2803.864495] LNet: Removed LNI 192.168.202.121@tcp [ 2804.283516] Key type .llcrypt unregistered [ 2804.284867] Key type ._llcrypt unregistered [ 2813.505161] Key type ._llcrypt registered [ 2813.506262] Key type .llcrypt registered [ 2813.545130] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 2813.907963] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2813.921177] alg: No test for adler32 (adler32-zlib) [ 2814.768754] Lustre: Lustre: Build Version: 2.17.57_103_g794a134 [ 2814.851570] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 2816.431234] Key type lgssc registered [ 2816.783322] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2819.358406] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2819.365376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2820.509766] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2821.897316] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2824.823408] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2824.959677] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2826.251823] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2828.693418] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2830.471057] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2832.625750] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2832.738153] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2834.624752] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2837.027522] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2846.054499] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 2850.271835] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2850.276273] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2850.282985] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2851.297970] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2851.298575] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2851.299090] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2851.299977] Lustre: Skipped 1 previous similar message [ 2852.447352] Lustre: lustre-OST0000: Not available for connect from 192.168.202.21@tcp (stopping) [ 2855.629233] Lustre: server umount lustre-OST0000 complete [ 2857.954244] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9bd0c3e5c800.ost_server_uuid 50 [ 2858.519132] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 2868.488141] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2868.489073] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2868.491759] Lustre: Skipped 1 previous similar message [ 2873.824119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2873.826635] Lustre: Skipped 2 previous similar messages [ 2874.645977] Lustre: server umount lustre-MDT0000 complete [ 2875.824164] LustreError: 192120:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788715105 with bad export cookie 11407945236565290607 [ 2875.827103] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2875.827971] LustreError: 192120:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2875.942612] Lustre: server umount lustre-MDT0001 complete [ 2878.569107] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 13:18:28 (1788715108) [ 2880.703427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2880.843426] LustreError: 192989:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2880.848600] LustreError: 192989:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2880.866140] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2882.126800] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2885.066377] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2885.190416] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2886.481953] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2888.762255] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2890.508829] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2892.847565] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2895.271614] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2897.576766] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2899.851526] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2899.934816] Lustre: lustre-OST0001: new disk, initializing [ 2899.936987] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2899.940866] Lustre: 195831:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2899.990769] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2899.992821] Lustre: Skipped 1 previous similar message [ 2901.850858] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2905.065787] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2905.069122] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2905.080899] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2905.523754] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2906.089886] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 2910.689426] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2910.692055] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2910.696676] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2912.443111] Lustre: server umount lustre-OST0000 complete [ 2914.339396] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2914.409568] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 2915.714339] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2915.783608] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 2915.929443] LustreError: 194810:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2921.049550] LustreError: 194813:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2921.054227] LustreError: 194813:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2926.560947] LustreError: 197288:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2926.562965] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2926.565193] LustreError: 197288:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2926.570240] Lustre: Skipped 2 previous similar messages [ 2931.681469] LustreError: 194813:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2931.682295] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2931.686371] LustreError: 194813:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2931.689013] Lustre: Skipped 6 previous similar messages [ 2931.858651] Lustre: server umount lustre-OST0001 complete [ 2936.288295] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2936.291916] LustreError: Skipped 1 previous similar message [ 2936.294170] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2939.232799] Lustre: server umount lustre-MDT0000 complete [ 2940.432238] LustreError: 192970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788715170 with bad export cookie 11407945236565291664 [ 2940.435853] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2940.438124] LustreError: 192970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2940.545400] Lustre: server umount lustre-MDT0001 complete [ 2948.067825] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 13:19:38 (1788715178) [ 2950.175398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2950.270131] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2950.281710] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2950.356152] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2951.748530] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2954.521287] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2954.552795] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2954.565218] Lustre: 199242:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2955.965941] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2958.321475] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2960.090456] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2962.269506] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2962.339672] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2964.263275] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2966.609017] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2967.580095] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 2968.813530] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2968.840265] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2970.832254] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2973.161489] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2974.179097] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2974.180689] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2974.183016] Lustre: Skipped 2 previous similar messages [ 2974.184830] Lustre: Skipped 2 previous similar messages [ 2975.756966] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 2980.096894] Lustre: server umount lustre-OST0000 complete [ 2980.835337] LustreError: 201088:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2980.840395] LustreError: 201088:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2991.078525] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2991.080691] LustreError: Skipped 1 previous similar message [ 2996.756086] Lustre: server umount lustre-OST0001 complete [ 3000.799854] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3004.339656] Lustre: server umount lustre-MDT0000 complete [ 3005.576646] LustreError: 202995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788715235 with bad export cookie 11407945236565293050 [ 3005.579766] LustreError: 202995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3005.581178] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3005.703594] Lustre: server umount lustre-MDT0001 complete [ 3013.398075] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 13:20:43 (1788715243) [ 3018.400212] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_hostid [ 3021.693154] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 3033.292420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3033.382151] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3033.391273] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3033.430859] Lustre: lustre-MDT0000: new disk, initializing [ 3033.450473] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3033.452256] Lustre: Skipped 3 previous similar messages [ 3033.457720] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3034.789431] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3039.019030] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3039.063209] Lustre: 207300:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3039.085955] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3039.088330] Lustre: Skipped 1 previous similar message [ 3039.124702] Lustre: lustre-MDT0001: new disk, initializing [ 3039.165851] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3039.168959] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3040.477277] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3044.033426] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3045.802919] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3047.986368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3048.114237] Lustre: lustre-OST0000: new disk, initializing [ 3048.115806] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3048.118068] Lustre: 209266:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3048.955267] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3048.958310] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3049.000754] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3050.140994] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3053.737503] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3055.097485] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3055.165462] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3056.473179] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3056.555402] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3059.171511] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3059.173418] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3059.177342] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3059.181241] Lustre: Skipped 5 previous similar messages [ 3059.189385] Lustre: Skipped 12 previous similar messages [ 3059.293644] Lustre: server umount lustre-OST0000 complete [ 3066.648739] Lustre: server umount lustre-MDT0000 complete [ 3067.835157] LustreError: 207293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788715297 with bad export cookie 11407945236565294793 [ 3067.839202] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3067.839620] LustreError: 207293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3067.949134] Lustre: server umount lustre-MDT0001 complete [ 3070.649218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3070.802776] LustreError: 211477:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3070.808804] LustreError: 211477:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 3072.097387] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3074.941983] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3076.427895] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3078.802902] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3080.709626] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3082.931962] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3085.031527] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3087.473090] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3088.900175] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3097.166457] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 8 sec [ 3098.624927] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3098.711421] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3100.640856] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3100.644317] LustreError: Skipped 1 previous similar message [ 3105.420411] Lustre: server umount lustre-OST0000 complete [ 3107.272160] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3108.721327] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3132.280561] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3132.391121] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3132.394291] Lustre: Skipped 5 previous similar messages [ 3134.337419] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3136.686556] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3137.193497] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3140.233521] Lustre: server umount lustre-OST0000 complete [ 3141.440502] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3141.445559] Lustre: Skipped 5 previous similar messages [ 3143.570964] Lustre: server umount lustre-MDT0000 complete [ 3144.787198] LustreError: 211458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788715374 with bad export cookie 11407945236565295661 [ 3144.791162] LustreError: 211458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3144.791456] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3144.909744] Lustre: server umount lustre-MDT0001 complete [ 3147.576476] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 13:22:57 (1788715377) [ 3149.682992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3149.833424] LustreError: 217486:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3151.150955] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3153.867648] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3155.440945] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3157.865297] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3159.743764] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3161.914744] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3163.920566] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3166.418239] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3167.854783] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3176.080601] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 8 sec [ 3177.485327] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3177.557507] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3179.689098] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3179.758889] Lustre: lustre-OST0001: new disk, initializing [ 3179.760992] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3179.763749] Lustre: 220909:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3181.690873] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3185.440638] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3186.703955] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3186.708405] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3186.748964] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3187.006729] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3187.095208] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3188.547290] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3188.632069] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3191.780991] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3191.784779] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3191.784875] LustreError: Skipped 2 previous similar messages [ 3191.787500] Lustre: Skipped 7 previous similar messages [ 3195.389369] Lustre: server umount lustre-OST0001 complete [ 3197.421318] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3197.496375] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3198.865720] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3198.940274] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3214.426285] LustreError: 222249:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3214.430951] LustreError: 222249:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 3222.549139] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3224.512409] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3226.939444] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3227.460487] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3227.621815] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3227.625260] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3227.625264] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3234.167135] Lustre: server umount lustre-OST0001 complete [ 3252.299996] Lustre: server umount lustre-OST0000 complete [ 3259.741251] Lustre: server umount lustre-MDT0000 complete [ 3261.004139] LustreError: 217469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788715490 with bad export cookie 11407945236565296592 [ 3261.008802] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3261.009151] LustreError: 217469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3267.184417] Lustre: server umount lustre-MDT0001 complete [ 3275.089048] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 13:25:05 (1788715505) [ 3277.312815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3277.411600] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3277.423494] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3277.503777] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3277.507123] Lustre: Skipped 5 previous similar messages [ 3278.870497] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3281.769781] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3281.807699] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3281.818287] Lustre: 226143:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3283.230521] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3285.437336] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3287.187424] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3289.250605] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3289.309065] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3291.109642] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3293.359693] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3296.856344] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3298.744231] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3300.945741] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3301.418304] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3302.671887] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3302.739731] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3304.039591] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3304.113851] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3305.784700] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9bd0d815b800.ost_server_uuid 50 [ 3306.263953] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9bd0d815b800.ost_server_uuid in FULL state after 0 sec [ 3306.586810] Lustre: Permanently deactivating lustre-OST0001 [ 3306.591948] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3306.674382] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 3307.346744] Lustre: Permanently reactivating lustre-OST0001 [ 3312.109210] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3312.113626] Lustre: Skipped 8 previous similar messages [ 3314.361457] Lustre: server umount lustre-OST0001 complete [ 3321.612956] Lustre: server umount lustre-OST0000 complete [ 3322.847720] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3322.849806] LustreError: Skipped 4 previous similar messages [ 3332.575138] LustreError: 230710:0:(osp_dev.c:699:osp_process_config()) lustre-OST0001-osc-MDT0001: unknown param osc.active=1 [ 3337.183122] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3337.244055] Lustre: server umount lustre-MDT0000 complete [ 3338.405476] LustreError: 226133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788715567 with bad export cookie 11407945236565297866 [ 3338.407914] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3338.409886] LustreError: 226133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3338.522186] Lustre: server umount lustre-MDT0001 complete [ 3345.546499] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 13:26:15 (1788715575) [ 3347.762194] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3347.860887] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3347.872144] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3347.876448] Lustre: Skipped 5 previous similar messages [ 3349.268897] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3352.010972] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3352.046066] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3352.049214] Lustre: Skipped 1 previous similar message [ 3352.059338] Lustre: 232832:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3353.521204] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3355.917686] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3357.743729] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3359.722840] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3359.793667] Lustre: Permanently deactivating lustre-OST0000 [ 3359.798292] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 3361.534252] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3363.656569] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3365.665116] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3367.393624] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3367.672518] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3367.675213] Lustre: Skipped 1 previous similar message [ 3367.715053] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 3369.496254] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3369.977771] Lustre: Permanently reactivating lustre-OST0000 [ 3375.095171] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 3375.098984] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3375.102743] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3375.104951] Lustre: Skipped 1 previous similar message [ 3375.120410] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 3375.128950] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3385.019693] Lustre: server umount lustre-OST0001 complete [ 3388.387130] LustreError: 234677:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3388.391624] LustreError: 234677:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 14 previous similar messages [ 3392.401814] Lustre: server umount lustre-OST0000 complete [ 3393.788970] Lustre: server umount lustre-MDT0000 complete [ 3394.994433] LustreError: 232823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788715624 with bad export cookie 11407945236565299588 [ 3394.998943] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3401.203121] Lustre: server umount lustre-MDT0001 complete [ 3404.768066] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 3405.836459] Key type lgssc unregistered [ 3405.960342] LNet: 237678:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3405.963713] LNetError: 237678:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3405.973330] LNet: Removed LNI 192.168.202.121@tcp [ 3406.283942] Key type .llcrypt unregistered [ 3406.285374] Key type ._llcrypt unregistered [ 3414.914248] Key type ._llcrypt registered [ 3414.915377] Key type .llcrypt registered [ 3414.952748] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 3415.292649] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3415.300336] alg: No test for adler32 (adler32-zlib) [ 3416.156499] Lustre: Lustre: Build Version: 2.17.57_103_g794a134 [ 3416.250097] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 3417.839205] Key type lgssc registered [ 3418.231862] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3421.595517] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3421.602809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3422.753114] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3424.164809] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3426.922799] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3426.958359] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3426.970765] Lustre: 239106:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3426.976316] Lustre: Permanently deactivating lustre-MDT0001 [ 3426.981864] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3427.096431] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3428.408561] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3429.036710] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3429.039525] LustreError: 239149:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3429.042826] LustreError: 239149:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3429.045851] LustreError: 239149:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 6, retries 1, failed: rc = -108 [ 3430.742421] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3432.636484] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3434.825786] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3434.937450] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3436.775294] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3436.964406] LustreError: 240960:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3436.967587] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3436.970083] LustreError: 240960:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3439.063445] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3441.400619] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3441.474356] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3443.309264] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3445.596479] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3446.758204] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 3447.785077] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 3447.785215] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 3451.369756] Lustre: Permanently reactivating lustre-MDT0001 [ 3451.375856] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 3451.377757] Lustre: Skipped 2 previous similar messages [ 3452.574919] Lustre: lustre-MDT0001: Client 5a42f851-5b30-4b0c-87ae-25de2899614e (at 192.168.202.21@tcp) reconnecting [ 3455.176097] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3455.653984] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3457.308652] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 3457.379877] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 3458.735854] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 3458.817279] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 3459.660559] Lustre: Permanently deactivating lustre-MDT0001 [ 3459.666911] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3459.670310] Lustre: Skipped 2 previous similar messages [ 3461.053442] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3463.139094] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3463.140795] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3463.141785] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3463.143736] Lustre: Skipped 1 previous similar message [ 3468.256760] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3468.259152] Lustre: Skipped 5 previous similar messages [ 3469.270580] Lustre: server umount lustre-MDT0000 complete [ 3470.518347] LustreError: 239098:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788715700 with bad export cookie 17147526994714163914 [ 3470.522417] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3470.523154] LustreError: 239098:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3470.929240] Lustre: server umount lustre-MDT0001 complete [ 3478.267977] Lustre: server umount lustre-OST0000 complete [ 3481.915903] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 13:28:32 (1788715712) [ 3485.428354] Lustre: server umount lustre-OST0001 complete [ 3487.501693] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_hostid [ 3490.651880] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 3502.143491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3502.227408] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3502.231557] Lustre: Skipped 2 previous similar messages [ 3502.239904] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3502.282834] Lustre: lustre-MDT0000: new disk, initializing [ 3502.304313] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3502.312296] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3503.647772] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3507.771670] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3507.817808] Lustre: 247361:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3507.842343] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3507.844038] Lustre: Skipped 1 previous similar message [ 3507.884376] Lustre: lustre-MDT0001: new disk, initializing [ 3507.927769] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3507.931347] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3509.403587] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3513.187337] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3514.988262] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3517.208740] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3517.311611] Lustre: lustre-OST0000: new disk, initializing [ 3517.313279] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3517.316241] Lustre: 249327:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3517.343982] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3517.345887] Lustre: Skipped 1 previous similar message [ 3517.746039] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3517.750112] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3517.781642] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3519.206415] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3522.919652] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3523.559086] LustreError: 247367:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3527.770331] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3527.834102] Lustre: lustre-OST0001: new disk, initializing [ 3527.835812] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3527.838479] Lustre: 250445:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3529.772212] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3533.518353] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3533.647106] LustreError: 247367:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3533.656100] LustreError: 249335:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3534.040581] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3535.871321] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3535.876856] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3535.903648] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3543.751054] LustreError: 249335:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3546.087294] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3546.087554] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3546.091461] Lustre: Skipped 2 previous similar messages [ 3546.092487] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3546.095371] LustreError: Skipped 1 previous similar message [ 3550.463199] Lustre: server umount lustre-OST0001 complete [ 3551.203771] LustreError: 249420:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3551.207716] LustreError: 249420:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3556.322964] LustreError: 251417:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3556.327938] LustreError: 251417:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3561.448129] LustreError: 249312:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3561.452776] LustreError: 249312:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3564.000146] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3564.000250] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3564.002363] LustreError: Skipped 1 previous similar message [ 3564.002912] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3564.006856] Lustre: Skipped 2 previous similar messages [ 3564.011012] Lustre: Skipped 3 previous similar messages [ 3566.563521] LustreError: 251214:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3566.570078] LustreError: 251214:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3568.596502] Lustre: server umount lustre-OST0000 complete [ 3571.681042] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3571.681653] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3571.686138] Lustre: Skipped 1 previous similar message [ 3571.688880] Lustre: Skipped 2 previous similar messages [ 3576.158117] Lustre: server umount lustre-MDT0000 complete [ 3576.800080] LustreError: 247371:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3576.805115] LustreError: 247371:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3577.393349] LustreError: 247353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788715806 with bad export cookie 17147526994714166728 [ 3577.396968] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3577.397773] LustreError: 247353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3583.598336] Lustre: server umount lustre-MDT0001 complete [ 3587.318810] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 3588.460497] Key type lgssc unregistered [ 3588.593367] LNet: 252477:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3588.596557] LNetError: 252477:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3588.608334] LNet: Removed LNI 192.168.202.121@tcp [ 3588.933533] Key type .llcrypt unregistered [ 3588.935170] Key type ._llcrypt unregistered [ 3602.903048] Key type ._llcrypt registered [ 3602.904303] Key type .llcrypt registered [ 3602.943310] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 3603.283684] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3603.320380] alg: No test for adler32 (adler32-zlib) [ 3604.183705] Lustre: Lustre: Build Version: 2.17.57_103_g794a134 [ 3604.277364] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 3605.855127] Key type lgssc registered [ 3606.217511] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3608.724609] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3608.731699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3609.816575] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3609.823801] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3609.884840] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3611.261055] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3614.016838] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3614.053206] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3614.064882] Lustre: 254656:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3614.158072] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3615.557498] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3617.965728] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3619.768392] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3622.029991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3622.099911] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3622.170046] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3623.444572] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 3624.109656] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3626.658408] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3669.475136] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3669.477716] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3669.478691] LustreError: Skipped 1 previous similar message [ 3669.479488] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3669.484021] Lustre: Skipped 1 previous similar message [ 3672.313438] Lustre: server umount lustre-OST0000 complete [ 3674.189587] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 3679.168637] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3679.340794] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3681.701716] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3684.544715] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3689.572235] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3689.577185] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3689.577306] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3689.586362] Lustre: Skipped 1 previous similar message [ 3720.673434] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3720.673926] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3720.674643] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3720.674648] Lustre: Skipped 1 previous similar message [ 3720.675993] LustreError: Skipped 1 previous similar message [ 3720.680722] Lustre: Skipped 1 previous similar message [ 3724.437868] Lustre: server umount lustre-OST0000 complete [ 3725.792257] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3725.792826] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3725.796650] Lustre: Skipped 1 previous similar message [ 3725.799386] Lustre: Skipped 2 previous similar messages [ 3730.913426] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3730.916607] Lustre: Skipped 1 previous similar message [ 3731.742594] Lustre: server umount lustre-MDT0000 complete [ 3733.122441] LustreError: 254692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788715962 with bad export cookie 10738285543910010656 [ 3733.126543] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3733.126692] LustreError: 254692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3733.253517] Lustre: server umount lustre-MDT0001 complete [ 3737.056758] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 3738.115240] Key type lgssc unregistered [ 3738.245410] LNet: 260069:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3738.248262] LNetError: 260069:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3738.259297] LNet: Removed LNI 192.168.202.121@tcp [ 3738.561984] Key type .llcrypt unregistered [ 3738.563676] Key type ._llcrypt unregistered [ 3747.522772] Key type ._llcrypt registered [ 3747.524216] Key type .llcrypt registered [ 3747.560715] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 3747.866949] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3747.879974] alg: No test for adler32 (adler32-zlib) [ 3748.739263] Lustre: Lustre: Build Version: 2.17.57_103_g794a134 [ 3748.832808] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 3750.415212] Key type lgssc registered [ 3750.824035] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3753.455996] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3753.463037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3754.622201] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3756.004109] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3758.857477] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3758.973481] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3760.265843] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3762.545663] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3764.303424] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3766.428319] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3766.530048] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3768.346920] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3770.628323] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3779.874374] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 3789.279911] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3789.283203] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3789.288292] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3790.305741] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3790.305932] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3790.306457] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3790.314148] Lustre: Skipped 1 previous similar message [ 3794.504052] Lustre: server umount lustre-OST0000 complete [ 3800.031892] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3800.034686] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3800.039772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3805.663927] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3805.665846] Lustre: Skipped 2 previous similar messages [ 3810.271079] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3810.326943] Lustre: server umount lustre-MDT0000 complete [ 3810.783984] LustreError: 261458:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3810.789861] LustreError: 261458:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3811.558268] LustreError: 263299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788716041 with bad export cookie 3344623008898050207 [ 3811.561091] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3811.563042] LustreError: 263299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3811.671112] Lustre: server umount lustre-MDT0001 complete [ 3815.200853] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 3816.170645] Key type lgssc unregistered [ 3816.294567] LNet: 265781:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3816.297217] LNetError: 265781:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3816.306868] LNet: Removed LNI 192.168.202.121@tcp [ 3816.614199] Key type .llcrypt unregistered [ 3816.615500] Key type ._llcrypt unregistered [ 3823.590738] Key type ._llcrypt registered [ 3823.592202] Key type .llcrypt registered [ 3823.627855] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 3824.095237] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3824.106902] alg: No test for adler32 (adler32-zlib) [ 3824.957184] Lustre: Lustre: Build Version: 2.17.57_103_g794a134 [ 3825.041302] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 3826.615163] Key type lgssc registered [ 3827.002088] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3829.671627] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3829.679306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3830.815837] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3832.147204] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3834.982481] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3835.099789] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3836.458723] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3838.981758] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3840.889667] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3843.177106] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3843.292645] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3845.266714] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3847.789330] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3856.613908] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 3866.080409] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3866.082673] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3866.086038] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3867.105185] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3867.106512] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3867.108270] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3868.243719] Lustre: server umount lustre-OST0000 complete [ 3871.199694] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3871.202285] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3871.206591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3871.208212] Lustre: Skipped 1 previous similar message [ 3875.604680] Lustre: server umount lustre-MDT0000 complete [ 3876.766091] LustreError: 267002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788716106 with bad export cookie 3550439778152217575 [ 3876.768953] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3876.769065] LustreError: 267002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3876.873995] Lustre: server umount lustre-MDT0001 complete [ 3880.084239] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 3881.008129] Key type lgssc unregistered [ 3881.116357] LNet: 270669:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3881.118438] LNetError: 270669:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3881.132292] LNet: Removed LNI 192.168.202.121@tcp [ 3881.401255] Key type .llcrypt unregistered [ 3881.402487] Key type ._llcrypt unregistered [ 3889.938447] Key type ._llcrypt registered [ 3889.940016] Key type .llcrypt registered [ 3889.979945] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 3890.270123] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3890.290625] alg: No test for adler32 (adler32-zlib) [ 3891.160194] Lustre: Lustre: Build Version: 2.17.57_103_g794a134 [ 3891.237585] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 3892.815188] Key type lgssc registered [ 3893.148926] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3895.574676] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3895.581092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3896.706166] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3897.965167] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3900.698739] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3900.827054] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3902.152728] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3904.660429] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3906.488714] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3908.727695] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3908.836863] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3910.727412] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3913.002306] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3922.150867] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3931.616441] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3931.618300] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3931.621536] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3932.641066] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3932.641907] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3932.643154] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3932.647659] Lustre: Skipped 1 previous similar message [ 3936.653881] Lustre: server umount lustre-OST0000 complete [ 3941.855947] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3941.858902] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3941.863637] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3947.999911] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3948.001236] Lustre: Skipped 3 previous similar messages [ 3952.095128] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3952.151075] Lustre: server umount lustre-MDT0000 complete [ 3953.120128] LustreError: 272057:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3953.127660] LustreError: 272057:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3953.390583] LustreError: 272038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788716182 with bad export cookie 16932046085605756829 [ 3953.393169] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3953.395210] LustreError: 272038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3953.517455] Lustre: server umount lustre-MDT0001 complete [ 3957.285919] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 3958.258983] Key type lgssc unregistered [ 3958.370366] LNet: 276382:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3958.372395] LNetError: 276382:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3958.379431] LNet: Removed LNI 192.168.202.121@tcp [ 3958.691106] Key type .llcrypt unregistered [ 3958.692204] Key type ._llcrypt unregistered [ 3966.011059] Key type ._llcrypt registered [ 3966.012377] Key type .llcrypt registered [ 3966.050444] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 3966.516535] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3966.566670] alg: No test for adler32 (adler32-zlib) [ 3967.454793] Lustre: Lustre: Build Version: 2.17.57_103_g794a134 [ 3967.556891] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 3969.143228] Key type lgssc registered [ 3969.542872] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3972.025610] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3972.033467] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3973.183408] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3974.442863] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3977.340188] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3977.462343] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3978.869193] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3981.286509] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3983.157900] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3985.488955] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3985.602819] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3987.463948] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3989.812584] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3997.927149] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 4008.419121] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4008.419195] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4008.425424] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4008.427256] Lustre: Skipped 1 previous similar message [ 4009.808053] Lustre: server umount lustre-OST0000 complete [ 4013.536879] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4013.536984] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4013.538057] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4013.540554] LustreError: Skipped 1 previous similar message [ 4013.544908] Lustre: Skipped 2 previous similar messages [ 4018.656312] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4018.658651] Lustre: Skipped 2 previous similar messages [ 4023.775956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4023.777598] Lustre: Skipped 1 previous similar message [ 4025.311077] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4025.366863] Lustre: server umount lustre-MDT0000 complete [ 4026.559326] LustreError: 277559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788716256 with bad export cookie 11103328635411150493 [ 4026.563458] LustreError: 277559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4026.563682] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4026.690843] Lustre: server umount lustre-MDT0001 complete [ 4030.011278] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 4030.930611] Key type lgssc unregistered [ 4031.062351] LNet: 281305:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4031.064528] LNetError: 281305:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4031.077285] LNet: Removed LNI 192.168.202.121@tcp [ 4031.359825] Key type .llcrypt unregistered [ 4031.361390] Key type ._llcrypt unregistered [ 4034.696708] Key type ._llcrypt registered [ 4034.697566] Key type .llcrypt registered [ 4034.730896] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 4039.898137] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_hostid [ 4045.787314] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 4046.115674] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4046.146217] alg: No test for adler32 (adler32-zlib) [ 4046.992703] Lustre: Lustre: Build Version: 2.17.57_103_g794a134 [ 4047.082879] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 4048.663142] Key type lgssc registered [ 4049.013713] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4059.951914] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4059.956874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4061.036804] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4061.046475] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4061.080172] Lustre: lustre-MDT0000: new disk, initializing [ 4061.099411] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4061.105225] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4062.416926] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4066.473918] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4066.512657] Lustre: 284825:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4066.534336] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4066.536017] Lustre: Skipped 1 previous similar message [ 4066.568387] Lustre: lustre-MDT0001: new disk, initializing [ 4066.597442] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4066.608054] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4066.610941] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4067.912151] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4071.541079] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4073.286701] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4075.439869] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4075.551979] Lustre: lustre-OST0000: new disk, initializing [ 4075.554114] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4075.557490] Lustre: 286793:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4075.588652] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4077.510819] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4081.129254] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4081.144753] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4081.148900] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4081.182316] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4082.476253] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4082.546394] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4083.910234] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4083.982565] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4086.242592] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4086.242683] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4086.250274] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4086.252180] Lustre: Skipped 1 previous similar message [ 4090.709546] Lustre: server umount lustre-OST0000 complete [ 4092.383888] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4092.386334] LustreError: Skipped 1 previous similar message [ 4092.387935] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4092.393336] Lustre: Skipped 1 previous similar message [ 4092.395627] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4096.480167] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4096.480593] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4101.600061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4101.601784] Lustre: Skipped 2 previous similar messages [ 4105.696410] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4105.699216] Lustre: Skipped 1 previous similar message [ 4106.207113] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4106.265575] Lustre: server umount lustre-MDT0000 complete [ 4107.483959] LustreError: 284818:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788716337 with bad export cookie 5862283511422177263 [ 4107.487416] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4107.487790] LustreError: 284818:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4107.599323] Lustre: server umount lustre-MDT0001 complete [ 4110.285619] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 13:39:00 (1788716340) [ 4112.366293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4112.528939] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4113.786585] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4116.505872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4116.624899] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4117.809077] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4120.010146] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4121.649986] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4123.646343] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4125.623264] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4128.065681] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4142.560430] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4142.563121] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4142.566536] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4147.534376] Lustre: server umount lustre-OST0000 complete [ 4148.705564] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4148.710413] Lustre: Skipped 2 previous similar messages [ 4154.963442] Lustre: server umount lustre-MDT0000 complete [ 4156.186331] LustreError: 289185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788716385 with bad export cookie 5862283511422178145 [ 4156.189569] LustreError: 289185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4156.189882] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4156.308601] Lustre: server umount lustre-MDT0001 complete [ 4159.796619] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 4160.782567] Key type lgssc unregistered [ 4160.906284] LNet: 292750:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4160.908143] LNetError: 292750:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4160.914252] LNet: Removed LNI 192.168.202.121@tcp [ 4161.214049] Key type .llcrypt unregistered [ 4161.215169] Key type ._llcrypt unregistered [ 4169.581472] Key type ._llcrypt registered [ 4169.583029] Key type .llcrypt registered [ 4169.617789] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_hostid [ 4175.587736] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 4175.964510] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4175.976490] alg: No test for adler32 (adler32-zlib) [ 4176.843179] Lustre: Lustre: Build Version: 2.17.57_103_g794a134 [ 4176.950198] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 4178.543171] Key type lgssc registered [ 4179.147566] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4207.428081] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4207.443186] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4208.582139] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4208.597960] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4208.651898] Lustre: lustre-MDT0000: new disk, initializing [ 4208.706341] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4208.717234] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4211.937435] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4219.198462] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4219.271621] Lustre: 296546:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4219.312763] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4219.316695] Lustre: Skipped 1 previous similar message [ 4219.366917] Lustre: lustre-MDT0001: new disk, initializing [ 4219.437440] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4219.467939] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4219.474327] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4222.077915] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4231.601853] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4235.311316] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4241.127721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4241.509747] Lustre: lustre-OST2710: new disk, initializing [ 4241.515609] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 4241.523236] Lustre: 298522:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4241.663301] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 4247.865415] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4248.670947] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 4248.694932] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 4248.850093] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 4256.985713] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4263.106649] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4263.304609] Lustre: lustre-OST03e8: new disk, initializing [ 4263.312078] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 4263.338670] Lustre: 299586:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4263.415413] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 4267.613468] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4269.607062] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 4269.611953] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 4269.650264] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 4271.394578] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4275.023108] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 4275.153032] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4277.261747] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 4277.369763] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4279.176331] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 4279.266429] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4281.058338] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 4281.170411] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4284.900296] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4284.904462] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4284.911016] Lustre: Skipped 3 previous similar messages [ 4284.915991] Lustre: Skipped 3 previous similar messages [ 4288.699405] Lustre: server umount lustre-MDT0000 complete [ 4290.320295] LustreError: 297349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788716519 with bad export cookie 15432891495182332713 [ 4290.323610] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4290.327311] LustreError: 297349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4290.457838] Lustre: server umount lustre-MDT0001 complete [ 4302.341057] Lustre: server umount lustre-OST2710 complete [ 4314.295802] Lustre: server umount lustre-OST03e8 complete [ 4322.372694] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_hostid [ 4326.814643] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 4342.647270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4342.769131] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4342.784676] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4342.822905] Lustre: lustre-MDT0000: new disk, initializing [ 4342.861908] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4342.878043] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4344.898349] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4350.988529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4351.053678] Lustre: 304827:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4351.087586] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4351.090660] Lustre: Skipped 1 previous similar message [ 4351.141624] Lustre: lustre-MDT0001: new disk, initializing [ 4351.191395] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4351.208549] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4351.219125] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4353.537682] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4359.206341] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4362.108363] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4365.741537] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4365.902937] Lustre: 306794:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4369.087423] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4374.895647] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4375.057301] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4375.063639] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4375.131037] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4377.076617] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4377.202289] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4379.425211] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4379.538587] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4385.253016] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4385.255319] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4385.260174] LustreError: Skipped 1 previous similar message [ 4385.260903] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4385.271581] Lustre: Skipped 1 previous similar message [ 4385.285986] Lustre: Skipped 1 previous similar message [ 4386.654336] Lustre: server umount lustre-OST0000 complete [ 4390.374202] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4390.383217] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4390.390912] Lustre: Skipped 1 previous similar message [ 4394.659958] Lustre: server umount lustre-MDT0000 complete [ 4395.491049] LustreError: 304837:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4395.497338] LustreError: 304837:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 4396.323705] LustreError: 305622:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788716625 with bad export cookie 15432891495182334393 [ 4396.325757] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4396.329323] LustreError: 305622:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4396.474840] Lustre: server umount lustre-MDT0001 complete [ 4400.889503] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 13:43:50 (1788716630) [ 4401.717553] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 4402.623338] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 13:43:52 (1788716632) [ 4404.317001] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 4418.359967] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4418.491215] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4418.504701] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4418.588710] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4418.593343] Lustre: Skipped 1 previous similar message [ 4420.506872] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4424.311624] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4424.360114] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4424.376234] Lustre: 310642:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4426.291435] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4443.303215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4443.410704] Lustre: Denying initial registration attempt from nid 192.168.202.121@tcp, specified as failover [ 4443.416065] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 4443.420104] LustreError: 312096:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -99 [ 4443.423752] LustreError: 312096:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 4443.426548] LustreError: 312096:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 4443.458461] Lustre: server umount lustre-OST0000 complete [ 4443.460427] LustreError: 312096:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 4450.271828] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4450.275342] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4450.280817] Lustre: Skipped 1 previous similar message [ 4450.284028] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4452.613290] Lustre: server umount lustre-MDT0000 complete [ 4454.664501] LustreError: 311384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788716684 with bad export cookie 15432891495182335275 [ 4454.669762] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4454.674986] LustreError: 311384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4454.869641] Lustre: server umount lustre-MDT0001 complete [ 4460.608512] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 4462.028825] Key type lgssc unregistered [ 4462.200478] LNet: 313215:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4462.204992] LNetError: 313215:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4462.219022] LNet: Removed LNI 192.168.202.121@tcp [ 4462.669300] Key type .llcrypt unregistered [ 4462.671354] Key type ._llcrypt unregistered [ 4467.970976] Key type ._llcrypt registered [ 4467.972682] Key type .llcrypt registered [ 4468.039707] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 4468.463846] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4468.519864] alg: No test for adler32 (adler32-zlib) [ 4469.452599] Lustre: Lustre: Build Version: 2.17.57_103_g794a134 [ 4469.578754] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 4471.199162] Key type lgssc registered [ 4471.814695] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4507.468912] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 4515.784551] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4515.867102] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4517.027941] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4517.046722] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4517.160613] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4522.807241] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4534.309546] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4534.555464] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4534.664227] Lustre: 315967:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4535.128852] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4541.038423] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4550.793941] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4559.658306] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4584.374554] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4584.522704] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4584.741515] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4590.175981] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4597.549931] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4602.848910] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4602.855346] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4602.855987] LustreError: Skipped 1 previous similar message [ 4602.892574] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4602.896542] Lustre: Skipped 1 previous similar message [ 4605.547552] Lustre: server umount lustre-OST0000 complete [ 4612.064994] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4612.071357] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4612.088314] Lustre: Skipped 1 previous similar message [ 4612.093907] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4613.089441] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4613.117196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4613.123510] Lustre: Skipped 1 previous similar message [ 4618.209639] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4618.215325] Lustre: Skipped 1 previous similar message [ 4623.327146] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4623.332795] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4623.356086] Lustre: Skipped 1 previous similar message [ 4623.522940] Lustre: server umount lustre-MDT0000 complete [ 4626.977849] LustreError: 317836:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788716856 with bad export cookie 5779943901797704655 [ 4626.983759] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4626.990187] LustreError: 317836:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4627.299934] Lustre: server umount lustre-MDT0001 complete [ 4638.485577] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 4641.622945] Key type lgssc unregistered [ 4641.917421] LNet: 319545:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4641.924784] LNetError: 319545:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4641.969877] LNet: Removed LNI 192.168.202.121@tcp [ 4642.940832] Key type .llcrypt unregistered [ 4642.943173] Key type ._llcrypt unregistered [ 4669.091237] Key type ._llcrypt registered [ 4669.092897] Key type .llcrypt registered [ 4669.202689] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 4670.756933] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4670.793884] alg: No test for adler32 (adler32-zlib) [ 4672.047237] Lustre: Lustre: Build Version: 2.17.57_103_g794a134 [ 4672.289550] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 4674.015172] Key type lgssc registered [ 4675.266851] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4683.029323] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4683.045042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4684.373041] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4689.093229] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4693.495251] LustreError: 320961:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4693.518929] LustreError: 320961:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4697.981951] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4698.378770] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4703.260674] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4710.472134] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4716.456158] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4725.506674] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4725.981200] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4732.456073] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4739.264811] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4753.901536] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4753.917994] Lustre: Skipped 2 previous similar messages [ 4753.920105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4753.935867] Lustre: Skipped 2 previous similar messages [ 4757.146154] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.21@tcp (stopping) [ 4759.010443] LustreError: 320956:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4759.027585] LustreError: 320956:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4759.090475] Lustre: server umount lustre-MDT0000 complete [ 4762.217034] LustreError: 323458:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4762.408934] LustreError: 320941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788716991 with bad export cookie 12639601370320147739 [ 4762.411445] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4762.417373] LustreError: 320941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4764.131095] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4764.149303] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4767.138826] LustreError: 320958:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4767.139062] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4767.165041] LustreError: 320958:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4769.042517] Lustre: server umount lustre-MDT0001 complete [ 4775.190259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4783.284535] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4787.620509] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xaf68e27a9479315a [ 4787.629864] Lustre: MGC192.168.202.121@tcp: Connection restored to 0@lo (at 0@lo) [ 4787.742063] LustreError: 324574:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4787.760845] LustreError: 324574:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4788.176548] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4788.220839] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 4793.701637] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4797.347070] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4797.350225] LustreError: 324575:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4797.362645] LustreError: 324575:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 4801.128631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4801.623518] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4806.646933] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4808.290869] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4822.356243] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4825.513858] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4834.814337] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4836.746650] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4844.512606] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4844.521986] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4844.542504] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4844.551023] Lustre: Skipped 1 previous similar message [ 4847.592438] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4847.601819] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4848.895372] Lustre: server umount lustre-OST0000 complete [ 4857.825961] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4857.830709] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4857.862882] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4857.862889] Lustre: Skipped 2 previous similar messages [ 4859.702325] Lustre: server umount lustre-MDT0000 complete [ 4862.959035] LustreError: 324574:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4862.982647] LustreError: 324574:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4864.253294] LustreError: 320942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788717093 with bad export cookie 12639601370320154970 [ 4864.262904] LustreError: 320942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4864.265530] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4864.592777] Lustre: server umount lustre-MDT0001 complete [ 4879.335634] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 4882.855713] Key type lgssc unregistered [ 4883.239156] LNet: 327250:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4883.250779] LNetError: 327250:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4883.276207] LNet: Removed LNI 192.168.202.121@tcp [ 4884.412419] Key type .llcrypt unregistered [ 4884.415095] Key type ._llcrypt unregistered [ 4909.477983] Key type ._llcrypt registered [ 4909.479691] Key type .llcrypt registered [ 4909.634710] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 4911.211739] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4911.295129] alg: No test for adler32 (adler32-zlib) [ 4912.510667] Lustre: Lustre: Build Version: 2.17.57_103_g794a134 [ 4912.857193] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 4914.591178] Key type lgssc registered [ 4915.841552] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4924.913272] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4924.951575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4926.552897] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4931.625843] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4936.675664] LustreError: 328663:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4936.716450] LustreError: 328663:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4941.720345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4941.812109] LustreError: 328664:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4942.124221] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4946.319539] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4954.733292] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4961.253436] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4966.885826] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4966.918402] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4966.925674] Lustre: Skipped 1 previous similar message [ 4967.971143] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4967.983336] Lustre: Skipped 1 previous similar message [ 4971.710977] Lustre: server umount lustre-MDT0000 complete [ 4973.025497] LustreError: 328658:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4973.046032] LustreError: 328658:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4975.150697] LustreError: 328645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788717204 with bad export cookie 6675104857779844889 [ 4975.152699] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4975.163727] LustreError: 328645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4975.391044] Lustre: server umount lustre-MDT0001 complete [ 4990.609995] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4990.780694] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4990.793529] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4990.885281] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4994.654505] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5004.420666] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5004.604711] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5004.693067] Lustre: 331251:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5005.069587] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5008.995778] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5016.612965] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5021.943034] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5032.380637] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5032.745419] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 5032.772075] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 5032.783316] LustreError: 333138:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -2 [ 5032.799736] LustreError: 333138:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 5032.811638] LustreError: 333138:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 5032.879728] Lustre: server umount lustre-OST0000 complete [ 5032.886245] LustreError: 333138:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5045.185418] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5045.484922] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5045.774771] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5051.936047] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5053.593687] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 5058.787398] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5065.964039] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5066.075678] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5066.228504] Lustre: lustre-OST0001: new disk, initializing [ 5066.232255] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5066.241617] Lustre: 334679:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5066.347920] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5072.024710] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5072.034082] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5072.125373] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5072.423860] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5080.484801] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5087.712942] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5087.722955] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5087.732044] Lustre: Skipped 1 previous similar message [ 5087.735849] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5087.743058] Lustre: Skipped 1 previous similar message [ 5089.683367] Lustre: server umount lustre-OST0001 complete [ 5092.857276] LustreError: 333606:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5092.883910] LustreError: 333606:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5094.374409] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5094.384563] LustreError: Skipped 1 previous similar message [ 5094.388017] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5094.399702] Lustre: Skipped 1 previous similar message [ 5094.403534] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5097.965680] LustreError: 334080:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5097.984405] LustreError: 334080:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5100.539910] Lustre: server umount lustre-OST0000 complete [ 5107.681143] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5107.683862] LustreError: Skipped 1 previous similar message [ 5107.686901] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5107.703414] Lustre: Skipped 1 previous similar message [ 5107.715932] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5107.722550] Lustre: Skipped 3 previous similar messages [ 5110.817669] Lustre: server umount lustre-MDT0000 complete [ 5113.313342] LustreError: 331264:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5113.351732] LustreError: 331264:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5115.372393] LustreError: 333620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788717344 with bad export cookie 6675104857779845442 [ 5115.374312] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5115.383317] LustreError: 333620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5115.738396] Lustre: server umount lustre-MDT0001 complete [ 5128.383594] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 5132.088984] Key type lgssc unregistered [ 5132.463038] LNet: 336637:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5132.478537] LNetError: 336637:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5132.502990] LNet: Removed LNI 192.168.202.121@tcp [ 5133.620242] Key type .llcrypt unregistered [ 5133.628204] Key type ._llcrypt unregistered [ 5215.690549] Key type ._llcrypt registered [ 5215.693171] Key type .llcrypt registered [ 5215.813579] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_hostid [ 5235.290298] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 5237.248286] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5237.283579] alg: No test for adler32 (adler32-zlib) [ 5238.566983] Lustre: Lustre: Build Version: 2.17.57_103_g794a134 [ 5239.024579] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 5240.815497] Key type lgssc registered [ 5242.618920] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5288.739664] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5288.780576] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5290.179361] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5290.222840] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5290.300966] Lustre: lustre-MDT0000: new disk, initializing [ 5290.429847] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5290.468305] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5296.457501] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5311.146545] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5311.326758] Lustre: 341576:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5311.397806] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5311.406933] Lustre: Skipped 1 previous similar message [ 5311.547840] Lustre: lustre-MDT0001: new disk, initializing [ 5311.667937] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5311.744020] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5311.759133] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5315.923395] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5328.766493] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5336.569436] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5346.340389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5346.766420] Lustre: lustre-OST0000: new disk, initializing [ 5346.771743] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5346.786479] Lustre: 343545:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5346.908883] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5356.698750] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5356.760847] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5356.786034] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5357.042279] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5368.625098] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5374.502943] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5374.734108] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5380.049725] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5380.309463] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5387.779361] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5387.786295] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5387.790643] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5387.795190] Lustre: Skipped 1 previous similar message [ 5387.831386] Lustre: Skipped 1 previous similar message [ 5392.875227] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5392.884167] Lustre: Skipped 1 previous similar message [ 5396.959170] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5397.235397] Lustre: server umount lustre-OST0000 complete [ 5403.107334] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5403.125912] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5403.130074] Lustre: Skipped 1 previous similar message [ 5403.158919] Lustre: Skipped 1 previous similar message [ 5408.045956] Lustre: server umount lustre-MDT0000 complete [ 5412.814725] LustreError: 341566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788717642 with bad export cookie 10588034642915794645 [ 5412.816521] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5412.822680] LustreError: 341566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5413.234171] Lustre: server umount lustre-MDT0001 complete [ 5425.183111] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 14:00:53 (1788717653) [ 5434.955922] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 14:01:04 (1788717664) [ 5445.431787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5445.991535] LustreError: 346317:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5446.016688] LustreError: 346317:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5446.097257] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5451.232933] LustreError: 346318:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5451.256837] LustreError: 346318:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5451.552988] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5456.364641] LustreError: 346317:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5461.094739] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5461.476382] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 5461.651503] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5467.135182] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5476.265688] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5483.347309] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5491.271533] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5491.749447] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5498.855138] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5508.168592] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5527.019376] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 5528.850666] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 5530.860173] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 5532.758215] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 5534.650437] Lustre: Failing over lustre-MDT0000 [ 5534.981466] Lustre: server umount lustre-MDT0000 complete [ 5535.345548] LustreError: 346312:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5536.753398] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5536.780036] Lustre: Skipped 2 previous similar messages [ 5540.452771] LustreError: 346314:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5540.475058] LustreError: 346314:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5550.688649] LustreError: 347907:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5550.707988] LustreError: 347907:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 5553.119176] Lustre: 339797:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788717766/real 1788717766] req@00000000bc0a2b78 x1875606207805440/t0(0) o400->MGC192.168.202.121@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788717782 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5553.160630] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5555.449073] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5563.371274] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x92f04168f086a661 [ 5563.398513] Lustre: MGC192.168.202.121@tcp: Connection restored to 0@lo (at 0@lo) [ 5563.740433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5563.829940] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5567.092629] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5569.014662] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5569.058657] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 5569.117162] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5569.140899] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5579.621372] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5582.040379] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5586.871507] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 5589.421575] Lustre: Failing over lustre-MDT0000 [ 5589.496667] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5589.509682] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5589.514795] LustreError: Skipped 1 previous similar message [ 5589.539734] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5589.550530] Lustre: Skipped 2 previous similar messages [ 5590.270799] Lustre: server umount lustre-MDT0000 complete [ 5592.722705] LustreError: 346312:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5592.758674] LustreError: 346312:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 15 previous similar messages [ 5610.463130] Lustre: 339794:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788717824/real 1788717824] req@00000000a9f56aa9 x1875606207827968/t0(0) o400->MGC192.168.202.121@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788717840 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5610.502378] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5612.091438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5620.707510] LustreError: 339793:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000006463ca3d x1875606207833216/t0(0) o250->MGC192.168.202.121@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5621.125181] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5621.181420] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5623.387933] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5625.843695] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5626.362483] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5626.370898] Lustre: Skipped 2 previous similar messages [ 5626.401392] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 5626.459053] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 5634.918579] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5637.041411] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5646.826486] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5646.826746] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5646.833563] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5646.840294] Lustre: Skipped 3 previous similar messages [ 5646.847893] Lustre: Skipped 1 previous similar message [ 5649.653108] Lustre: server umount lustre-MDT0000 complete [ 5651.949178] LustreError: 346317:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5651.977617] LustreError: 346317:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 29 previous similar messages [ 5654.215445] LustreError: 350309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788717883 with bad export cookie 10588034642915797368 [ 5654.223586] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5654.247802] LustreError: 350309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5654.905611] Lustre: server umount lustre-MDT0001 complete [ 5670.039395] Lustre: server umount lustre-OST0000 complete [ 5684.030523] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5689.341800] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5698.547507] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5699.023478] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5699.032358] Lustre: Skipped 1 previous similar message [ 5704.249736] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5713.843403] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5720.040285] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5727.464877] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5728.770902] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 5735.238944] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5744.466356] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5751.976178] Lustre: server umount lustre-OST0000 complete [ 5759.456366] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5759.460885] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5759.467617] Lustre: Skipped 2 previous similar messages [ 5759.479792] Lustre: Skipped 1 previous similar message [ 5769.631148] Lustre: 339794:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788717983/real 1788717983] req@000000002c56178d x1875606207879680/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788717999 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5770.207138] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5770.360760] Lustre: server umount lustre-MDT0000 complete [ 5774.763358] LustreError: 354528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788718004 with bad export cookie 10588034642915798110 [ 5774.772804] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5775.004699] Lustre: server umount lustre-MDT0001 complete [ 5784.541770] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 5787.502869] Key type lgssc unregistered [ 5787.935762] LNet: 356216:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5787.954121] LNetError: 356216:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5788.001785] LNet: Removed LNI 192.168.202.121@tcp [ 5789.499205] Key type .llcrypt unregistered [ 5789.500967] Key type ._llcrypt unregistered [ 5821.067649] Key type ._llcrypt registered [ 5821.072755] Key type .llcrypt registered [ 5821.236854] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 5822.634625] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5822.679247] alg: No test for adler32 (adler32-zlib) [ 5823.814965] Lustre: Lustre: Build Version: 2.17.57_103_g794a134 [ 5824.109560] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 5825.839226] Key type lgssc registered [ 5827.002694] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5834.978422] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5835.000801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5836.400721] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5841.299630] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5845.988144] LustreError: 357732:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5846.012610] LustreError: 357732:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5850.211327] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5850.608029] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5856.099643] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5864.954369] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5871.058690] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5878.250655] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5878.622331] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5880.725214] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 5884.655377] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5891.433767] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5911.526200] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5911.536442] Lustre: Skipped 1 previous similar message [ 5911.542336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5911.553113] Lustre: Skipped 2 previous similar messages [ 5913.824668] Lustre: server umount lustre-MDT0000 complete [ 5916.642060] LustreError: 358459:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5916.662374] LustreError: 358459:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5917.684458] LustreError: 358466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788718147 with bad export cookie 8521209598888893331 [ 5917.689733] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5917.698069] LustreError: 358466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5918.059390] Lustre: server umount lustre-MDT0001 complete [ 5932.273350] Lustre: server umount lustre-OST0000 complete [ 5945.378508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5945.859647] LustreError: 361463:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5945.962877] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5950.515549] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5951.463445] LustreError: 361464:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5955.580884] LustreError: 361463:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5959.383173] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5959.748919] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5964.505214] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5971.758637] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5977.749389] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5985.015560] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5985.192089] LustreError: 363285:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5985.451935] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5990.599433] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5992.540659] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6000.976188] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6015.968540] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6015.980671] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6015.990606] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6018.767283] Lustre: server umount lustre-MDT0000 complete [ 6021.606539] LustreError: 361463:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6021.648991] LustreError: 361463:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6022.322427] LustreError: 361445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788718251 with bad export cookie 8521209598888894444 [ 6022.340177] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6022.344500] LustreError: 361445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6022.608779] Lustre: server umount lustre-MDT0001 complete [ 6037.461994] Lustre: server umount lustre-OST0000 complete [ 6052.734164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6053.227321] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6057.889313] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6058.466298] LustreError: 365374:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6058.484856] LustreError: 365374:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6066.856302] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6073.510756] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6083.292701] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6091.738126] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6100.891605] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6101.203220] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6101.218269] Lustre: Skipped 1 previous similar message [ 6108.676585] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6117.002582] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6123.840477] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 6137.266255] Lustre: server umount lustre-OST0000 complete [ 6143.969244] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6143.988706] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6144.003572] Lustre: Skipped 2 previous similar messages [ 6144.015388] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6144.029941] Lustre: Skipped 3 previous similar messages [ 6149.602617] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6149.615449] Lustre: Skipped 3 previous similar messages [ 6151.794298] Lustre: server umount lustre-MDT0000 complete [ 6152.671934] Lustre: 357043:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788718366/real 1788718366] req@000000009bf510f4 x1875606821238656/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788718382 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6152.710446] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6152.723253] Lustre: Skipped 1 previous similar message [ 6152.737150] LustreError: 365374:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6152.755333] LustreError: 365374:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6154.739920] Lustre: 357042:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788718368/real 1788718368] req@0000000014efba34 x1875606821239680/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788718384 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6155.783870] LustreError: 365355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788718385 with bad export cookie 8521209598888895466 [ 6155.802023] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6156.276781] Lustre: server umount lustre-MDT0001 complete [ 6168.883607] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 6171.992228] Key type lgssc unregistered [ 6172.325932] LNet: 368926:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6172.340884] LNetError: 368926:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6172.381834] LNet: Removed LNI 192.168.202.121@tcp [ 6173.435594] Key type .llcrypt unregistered [ 6173.437660] Key type ._llcrypt unregistered [ 6199.521745] Key type ._llcrypt registered [ 6199.524662] Key type .llcrypt registered [ 6199.662412] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 6200.812324] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6200.836164] alg: No test for adler32 (adler32-zlib) [ 6202.015130] Lustre: Lustre: Build Version: 2.17.57_103_g794a134 [ 6202.283545] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 6204.016699] Key type lgssc registered [ 6205.225375] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6213.765057] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6213.802206] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6213.819323] LustreError: 370366:0:(osd_handler.c:8547:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 6213.856463] LustreError: 370366:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 6213.864131] LustreError: 370366:0:(obd_mount.c:261:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 6213.872192] LustreError: 370366:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 6213.877616] LustreError: 370366:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6221.775826] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6221.785777] LustreError: 370763:0:(osd_handler.c:8547:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 6221.811217] LustreError: 370763:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 6221.817199] LustreError: 370763:0:(obd_mount.c:261:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 6221.829968] LustreError: 370763:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 6221.841833] LustreError: 370763:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6241.002532] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 6243.659607] Key type lgssc unregistered [ 6243.932954] LNet: 371764:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6243.939389] LNetError: 371764:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6243.959715] LNet: Removed LNI 192.168.202.121@tcp [ 6244.823250] Key type .llcrypt unregistered [ 6244.826137] Key type ._llcrypt unregistered [ 6263.388916] Key type ._llcrypt registered [ 6263.395445] Key type .llcrypt registered [ 6263.540183] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_hostid [ 6280.437405] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 6282.112887] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6282.173682] alg: No test for adler32 (adler32-zlib) [ 6283.290451] Lustre: Lustre: Build Version: 2.17.57_103_g794a134 [ 6283.619436] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 6285.343193] Key type lgssc registered [ 6286.996468] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6323.232244] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6323.281617] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6324.832435] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6324.885802] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6324.995332] Lustre: lustre-MDT0000: new disk, initializing [ 6325.061619] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6325.070097] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6330.602693] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6341.624006] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6341.799531] Lustre: 374917:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6341.849907] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6341.854096] Lustre: Skipped 1 previous similar message [ 6341.914135] Lustre: lustre-MDT0001: new disk, initializing [ 6341.984117] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6342.006708] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6342.012430] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6346.305982] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6356.847763] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6363.186392] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6371.550706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6371.893576] Lustre: lustre-OST0000: new disk, initializing [ 6371.896919] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6371.901766] Lustre: 376890:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6371.961790] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6379.644740] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6380.103497] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6380.113962] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6380.285640] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6389.958234] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6394.959240] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6395.243917] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6400.096724] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6400.320960] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6406.112974] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6406.118544] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6406.135146] LustreError: Skipped 1 previous similar message [ 6406.136064] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6406.178189] Lustre: Skipped 1 previous similar message [ 6408.838732] Lustre: server umount lustre-OST0000 complete [ 6413.793769] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6413.806162] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6413.842663] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6413.848603] Lustre: Skipped 1 previous similar message [ 6416.353415] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6416.361934] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6419.306479] Lustre: server umount lustre-MDT0000 complete [ 6423.053226] LustreError: 376889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788718652 with bad export cookie 17885449107312358207 [ 6423.065305] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6423.067558] LustreError: 376889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6423.416192] Lustre: server umount lustre-MDT0001 complete [ 6432.335104] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 6433.810680] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 14:17:43 (1788718663) [ 6441.158384] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6441.556088] LustreError: 379349:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6441.583558] LustreError: 379349:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6441.664323] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6447.081952] LustreError: 379350:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6447.545549] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6452.192681] LustreError: 379349:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6457.324417] LustreError: 379350:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6457.492849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6457.865440] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6462.941900] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6473.033489] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6479.215843] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6486.275511] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6486.449348] LustreError: 381172:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6486.752163] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6494.103759] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6502.313625] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6509.651992] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6509.851383] Lustre: lustre-OST0001: new disk, initializing [ 6509.856685] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6509.862295] Lustre: 382187:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6509.979388] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6514.895422] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6518.959285] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6518.982236] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6519.163326] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6525.463229] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6532.241270] Lustre: lustre-OST0001: Not available for connect from 192.168.202.21@tcp (stopping) [ 6532.248690] Lustre: Skipped 1 previous similar message [ 6534.630232] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6534.635520] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6534.639170] Lustre: Skipped 1 previous similar message [ 6534.644084] LustreError: Skipped 1 previous similar message [ 6535.741829] Lustre: server umount lustre-OST0001 complete [ 6537.314560] LustreError: 381174:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6553.569108] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6553.572369] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6553.588891] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6553.594415] Lustre: Skipped 2 previous similar messages [ 6555.121161] LustreError: 381173:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6555.136875] LustreError: 381173:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 6559.020824] Lustre: server umount lustre-OST0000 complete [ 6565.348386] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6565.352775] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6565.365793] Lustre: Skipped 2 previous similar messages [ 6565.366195] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6565.373697] Lustre: Skipped 4 previous similar messages [ 6565.386452] LustreError: Skipped 1 previous similar message [ 6577.119140] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6577.301217] Lustre: server umount lustre-MDT0000 complete [ 6580.549053] LustreError: 379332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788718810 with bad export cookie 17885449107312359089 [ 6580.560131] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6580.560909] LustreError: 379332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6586.959862] Lustre: server umount lustre-MDT0001 complete [ 6596.677956] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 6599.264986] Key type lgssc unregistered [ 6599.783102] LNet: 384178:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6599.801349] LNetError: 384178:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6599.835087] LNet: Removed LNI 192.168.202.121@tcp [ 6601.020161] Key type .llcrypt unregistered [ 6601.024808] Key type ._llcrypt unregistered [ 6658.807696] Key type ._llcrypt registered [ 6658.811629] Key type .llcrypt registered [ 6658.902744] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 6660.168981] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6660.212559] alg: No test for adler32 (adler32-zlib) [ 6661.289550] Lustre: Lustre: Build Version: 2.17.57_103_g794a134 [ 6661.544044] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 6663.239170] Key type lgssc registered [ 6664.395651] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6671.997336] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6672.029923] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6673.408658] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6673.431297] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6673.616848] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6678.287387] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6687.095065] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6687.209979] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6687.249918] Lustre: 386977:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6687.660625] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6691.877616] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6698.852624] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6704.705664] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6708.192702] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6708.203124] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6708.225229] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6710.754761] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6710.762235] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6713.685423] Lustre: server umount lustre-MDT0000 complete [ 6718.091444] LustreError: 386967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788718947 with bad export cookie 9418450127869194824 [ 6718.097353] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6718.109511] LustreError: 386967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6718.447943] Lustre: server umount lustre-MDT0001 complete [ 6728.521708] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 14:22:37 (1788718957) [ 6735.189053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6735.893129] LustreError: 389487:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6735.904953] LustreError: 389487:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6736.006516] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6741.054611] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6741.476316] LustreError: 389488:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6746.596640] LustreError: 389487:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6749.658500] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6750.292461] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6755.774960] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6764.043350] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6770.292825] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6777.392513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6777.731266] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6778.063763] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6785.684260] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6794.228337] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6802.662803] Lustre: Permanently deactivating lustre-OST0000 [ 6802.689603] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6809.654990] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6812.222659] LustreError: 392475:0:(mgs_llog.c:2046:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 6812.251202] LustreError: 392475:0:(mgs_llog.c:2101:mgs_replace_nids()) Only MGS is allowed to be started [ 6812.261388] LustreError: 392475:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 6815.915383] LustreError: 392524:0:(mgs_llog.c:2101:mgs_replace_nids()) Only MGS is allowed to be started [ 6815.919411] LustreError: 392524:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 6825.084646] Lustre: server umount lustre-OST0000 complete [ 6828.314642] LustreError: 392773:0:(mgs_llog.c:2101:mgs_replace_nids()) Only MGS is allowed to be started [ 6828.322565] LustreError: 392773:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 6832.102630] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6832.109101] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6832.131859] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6832.140061] Lustre: Skipped 1 previous similar message [ 6834.146322] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6834.162444] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6839.266695] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6839.272623] Lustre: Skipped 2 previous similar messages [ 6841.417266] Lustre: server umount lustre-MDT0000 complete [ 6844.388137] LustreError: 389488:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6844.405801] LustreError: 389488:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6845.206692] LustreError: 389469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788719074 with bad export cookie 9418450127869195405 [ 6845.208749] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6845.214390] LustreError: 389469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6854.809854] Lustre: server umount lustre-MDT0001 complete [ 6863.061520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6867.287087] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6869.733328] LustreError: 393935:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 6875.387060] Lustre: 394274:0:(mgs_llog.c:1811:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.121@tcp [ 6877.262300] Lustre: server umount MGS complete [ 6888.453181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6889.043489] LustreError: 394998:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6889.068278] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6889.076132] Lustre: Skipped 1 previous similar message [ 6889.132704] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6893.475744] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6899.681790] LustreError: 394999:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6899.691812] LustreError: 394999:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6901.599027] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6901.939459] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6907.329493] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6916.834653] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6922.793959] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6924.832353] Lustre: Permanently reactivating lustre-OST0000 [ 6924.872970] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6924.891548] Lustre: Skipped 2 previous similar messages [ 6952.418034] LustreError: 396751:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 6960.128617] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6960.514720] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6960.522848] Lustre: Skipped 1 previous similar message [ 6966.729061] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6973.738584] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6982.114559] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6982.128363] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6982.158447] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6985.601890] Lustre: server umount lustre-OST0000 complete [ 6993.377463] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6993.387682] Lustre: Skipped 2 previous similar messages [ 6995.820650] Lustre: server umount lustre-MDT0000 complete [ 6998.498468] LustreError: 395726:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6998.518244] LustreError: 395726:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6999.511023] LustreError: 394980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788719229 with bad export cookie 9418450127869196448 [ 6999.517161] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6999.529611] LustreError: 394980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6999.818112] Lustre: server umount lustre-MDT0001 complete [ 7011.303494] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 7014.072959] Key type lgssc unregistered [ 7014.343833] LNet: 399358:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7014.350081] LNetError: 399358:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7015.406378] LNet: Removed LNI 192.168.202.121@tcp [ 7016.411924] Key type .llcrypt unregistered [ 7016.416883] Key type ._llcrypt unregistered [ 7033.376965] Key type ._llcrypt registered [ 7033.380427] Key type .llcrypt registered [ 7033.492716] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_hostid [ 7047.953821] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 7048.764059] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7048.780000] alg: No test for adler32 (adler32-zlib) [ 7049.860733] Lustre: Lustre: Build Version: 2.17.57_103_g794a134 [ 7050.149452] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 7051.759191] Key type lgssc registered [ 7052.791760] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7090.325559] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7090.362198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7091.819732] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7091.881333] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7092.116448] Lustre: lustre-MDT0000: new disk, initializing [ 7092.475159] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7092.504086] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7098.680955] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 7111.115930] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7111.300558] Lustre: 402514:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7111.413175] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7111.423047] Lustre: Skipped 1 previous similar message [ 7111.561730] Lustre: lustre-MDT0001: new disk, initializing [ 7111.695289] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7111.761130] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7111.768663] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7115.992219] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 7126.303924] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7131.767596] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7138.724101] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7138.924273] Lustre: lustre-OST0000: new disk, initializing [ 7138.927468] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7138.932271] Lustre: 404483:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7139.028788] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7145.455265] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 7147.588826] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7147.599270] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7147.717994] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7155.510873] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7160.681269] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7160.925687] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7165.791441] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7165.967106] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7168.480095] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7168.484101] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7168.495059] LustreError: Skipped 1 previous similar message [ 7168.504623] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7168.511138] Lustre: Skipped 1 previous similar message [ 7168.522902] Lustre: Skipped 1 previous similar message [ 7173.607093] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7173.616159] Lustre: Skipped 1 previous similar message [ 7178.735395] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7178.742000] Lustre: Skipped 1 previous similar message [ 7182.303180] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7182.413983] Lustre: server umount lustre-OST0000 complete [ 7188.449512] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7188.459652] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7188.472903] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7191.895818] Lustre: server umount lustre-MDT0000 complete [ 7195.878492] LustreError: 402508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788719425 with bad export cookie 6745641197646405569 [ 7195.882053] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7195.888702] LustreError: 402508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7196.296934] Lustre: server umount lustre-MDT0001 complete [ 7204.903800] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 7206.502219] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 14:30:35 (1788719435) [ 7214.052640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7214.476428] LustreError: 406948:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7214.492340] LustreError: 406948:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7214.551594] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7219.485679] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 7219.681150] LustreError: 406949:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7224.802587] LustreError: 406948:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7228.519585] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7228.913697] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7232.601166] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 7238.982981] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7244.014875] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7251.062474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7251.552184] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7257.086281] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 7263.888794] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7265.445310] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 7267.971030] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 7274.464503] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7274.475068] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7274.483414] Lustre: Skipped 1 previous similar message [ 7274.489319] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7274.492015] Lustre: Skipped 1 previous similar message [ 7279.125847] Lustre: server umount lustre-OST0000 complete [ 7285.217710] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7285.229906] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7285.250335] Lustre: Skipped 1 previous similar message [ 7285.263262] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7285.277869] Lustre: Skipped 2 previous similar messages [ 7289.787285] Lustre: server umount lustre-MDT0000 complete [ 7290.855540] LustreError: 406948:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7290.871193] LustreError: 406948:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7295.798133] LustreError: 406929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788719525 with bad export cookie 6745641197646406444 [ 7295.814149] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7295.817968] LustreError: 406929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7296.409075] Lustre: server umount lustre-MDT0001 complete [ 7307.225705] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 7309.409479] Key type lgssc unregistered [ 7309.693439] LNet: 410676:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7309.706982] LNetError: 410676:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7309.739661] LNet: Removed LNI 192.168.202.121@tcp [ 7310.549326] Key type .llcrypt unregistered [ 7310.551709] Key type ._llcrypt unregistered [ 7332.607468] Key type ._llcrypt registered [ 7332.609924] Key type .llcrypt registered [ 7332.783250] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 7333.120036] Key type .llcrypt unregistered [ 7333.132703] Key type ._llcrypt unregistered [ 7348.651372] Key type ._llcrypt registered [ 7348.652974] Key type .llcrypt registered [ 7348.756955] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 7349.756520] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7349.838413] alg: No test for adler32 (adler32-zlib) [ 7351.025938] Lustre: Lustre: Build Version: 2.17.57_103_g794a134 [ 7351.560610] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 7353.407277] Key type lgssc registered [ 7354.177082] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7363.288368] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7363.327381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7365.304348] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7370.283117] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 7374.820071] LustreError: 412873:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7374.842905] LustreError: 412873:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7378.261467] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7378.732643] LustreError: 412872:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7378.741761] LustreError: 412872:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7378.769785] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7384.033686] LustreError: 412873:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7384.043238] LustreError: 412873:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7384.710582] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 7389.156088] LustreError: 412872:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7389.172540] LustreError: 412872:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7390.202173] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 7391.290929] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7395.302854] LustreError: 412873:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7395.318594] LustreError: 412873:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7398.155304] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7398.535642] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7402.864837] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 7412.222930] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7412.234468] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7413.264877] Lustre: server umount lustre-OST0000 complete [ 7417.316541] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7417.323493] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7417.351716] Lustre: Skipped 2 previous similar messages [ 7417.364536] Lustre: Skipped 2 previous similar messages [ 7419.421324] Lustre: server umount lustre-MDT0000 complete [ 7422.446930] LustreError: 414216:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7422.465841] LustreError: 414216:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7422.713108] LustreError: 412854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788719652 with bad export cookie 9636502426744160752 [ 7422.723360] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7422.727739] LustreError: 412854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7423.064583] Lustre: server umount lustre-MDT0001 complete [ 7432.663374] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 7434.860982] Key type lgssc unregistered [ 7435.079611] LNet: 416016:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7435.088528] LNetError: 416016:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7435.118527] LNet: Removed LNI 192.168.202.121@tcp [ 7435.787869] Key type .llcrypt unregistered [ 7435.789715] Key type ._llcrypt unregistered [ 7457.236509] Key type ._llcrypt registered [ 7457.240622] Key type .llcrypt registered [ 7457.341803] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 7458.525540] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7458.570361] alg: No test for adler32 (adler32-zlib) [ 7459.727899] Lustre: Lustre: Build Version: 2.17.57_103_g794a134 [ 7459.990978] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 7461.664177] Key type lgssc registered [ 7462.951367] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7469.828671] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7469.853875] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7485.343602] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7490.463845] LustreError: 417403:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.121@tcp: failed processing log, type 4: rc = -110 [ 7517.215216] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7523.881559] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 7533.124889] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7540.986713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7541.725270] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7541.804709] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 7546.319097] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 7546.854950] LustreError: 417429:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7546.880726] LustreError: 417429:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7551.972585] LustreError: 417428:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7551.993130] LustreError: 417428:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7555.202256] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7555.749117] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7560.599228] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 7567.741787] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7573.606929] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7581.155916] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7581.165445] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7581.174428] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7581.687155] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7581.710431] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7582.862626] Lustre: server umount lustre-OST0000 complete [ 7588.843855] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 7588.852829] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7588.867439] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7589.315488] Lustre: server umount lustre-MDT0000 complete [ 7591.904538] LustreError: 418445:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7591.906617] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7592.758538] LustreError: 420222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788719822 with bad export cookie 13670539536666879346 [ 7592.759468] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7592.766116] LustreError: 420222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7593.076984] Lustre: server umount lustre-MDT0001 complete [ 7605.025474] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 7608.232566] Key type lgssc unregistered [ 7608.456561] LNet: 420973:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7608.460957] LNetError: 420973:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7608.471424] LNet: Removed LNI 192.168.202.121@tcp [ 7609.053587] Key type .llcrypt unregistered [ 7609.055710] Key type ._llcrypt unregistered [ 7635.065461] Key type ._llcrypt registered [ 7635.067784] Key type .llcrypt registered [ 7635.208399] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 7636.244429] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7636.384125] alg: No test for adler32 (adler32-zlib) [ 7637.597478] Lustre: Lustre: Build Version: 2.17.57_103_g794a134 [ 7637.961891] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 7639.655225] Key type lgssc registered [ 7640.761753] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7648.329974] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7648.347866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7649.842952] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7654.667306] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 7660.001740] LustreError: 422397:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7660.009853] LustreError: 422397:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7662.970776] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7663.436996] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7668.656401] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 7676.334796] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7682.575573] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7690.121421] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7690.520405] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7694.652744] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 7697.198522] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 7705.627586] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7709.668611] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7709.679634] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7709.693948] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7712.390039] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.21@tcp (stopping) [ 7713.255673] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7713.268866] Lustre: Skipped 1 previous similar message [ 7715.508819] Lustre: server umount lustre-MDT0000 complete [ 7717.859222] LustreError: 422377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.21@tcp arrived at 1788719947 with bad export cookie 7679848215934167529 [ 7718.368199] LustreError: 422398:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7718.388207] LustreError: 422398:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7719.735348] LustreError: 422378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788719949 with bad export cookie 7679848215934166640 [ 7719.740195] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7719.747479] LustreError: 422378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7723.490470] LustreError: 422394:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7723.491173] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7723.501030] LustreError: 422394:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7723.544198] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7723.560709] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7723.567773] Lustre: Skipped 3 previous similar messages [ 7726.004184] Lustre: server umount lustre-OST0000 complete [ 7729.637601] LustreError: 423125:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7729.650188] LustreError: 423125:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7733.236531] Lustre: server umount lustre-MDT0001 complete [ 7745.289941] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 7748.397189] Key type lgssc unregistered [ 7748.730950] LNet: 426081:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7748.744290] LNetError: 426081:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7748.771411] LNet: Removed LNI 192.168.202.121@tcp [ 7749.703955] Key type .llcrypt unregistered [ 7749.706169] Key type ._llcrypt unregistered [ 7775.150723] Key type ._llcrypt registered [ 7775.165207] Key type .llcrypt registered [ 7775.366865] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 7777.047069] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7777.440955] alg: No test for adler32 (adler32-zlib) [ 7778.606808] Lustre: Lustre: Build Version: 2.17.57_103_g794a134 [ 7778.876564] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 7780.552051] Key type lgssc registered [ 7782.418208] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7792.292202] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7792.309028] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7793.719128] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7797.944700] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 7803.875988] LustreError: 427493:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7803.905522] LustreError: 427493:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7805.352822] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7805.756932] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7810.214696] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 7816.967289] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7823.422883] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7829.875372] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7830.327904] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7838.549749] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 7838.733083] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 7847.738711] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7852.008214] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 7852.014164] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7852.028982] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7854.052749] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7854.056722] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7858.296919] Lustre: server umount lustre-MDT0001 complete [ 7859.170405] LustreError: 429279:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7859.195710] LustreError: 429279:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7864.305341] LustreError: 427490:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7864.340628] LustreError: 427490:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7869.413687] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7869.420184] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7869.425540] LustreError: 427494:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7869.436303] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7869.451232] Lustre: Skipped 2 previous similar messages [ 7871.203346] Lustre: server umount lustre-OST0000 complete [ 7874.533414] LustreError: 427493:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7877.877770] Lustre: server umount lustre-MDT0000 complete [ 7893.396266] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 7896.418684] Key type lgssc unregistered [ 7896.817670] LNet: 431178:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7896.836659] LNetError: 431178:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7896.863939] LNet: Removed LNI 192.168.202.121@tcp [ 7897.838384] Key type .llcrypt unregistered [ 7897.845843] Key type ._llcrypt unregistered [ 7927.004620] Key type ._llcrypt registered [ 7927.008573] Key type .llcrypt registered [ 7927.134071] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_hostid [ 7942.405320] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 7943.722378] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7943.759706] alg: No test for adler32 (adler32-zlib) [ 7944.803975] Lustre: Lustre: Build Version: 2.17.57_103_g794a134 [ 7944.999864] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 7946.655344] Key type lgssc registered [ 7947.954801] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7988.088435] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 8004.466112] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8004.482099] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8005.825412] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8005.854463] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8005.944395] Lustre: lustre-MDT0000: new disk, initializing [ 8006.046583] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8006.062932] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8010.637957] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 8027.500023] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8027.720564] Lustre: lustre-OST0000: new disk, initializing [ 8027.724693] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8027.726547] Lustre: Skipped 1 previous similar message [ 8027.730140] Lustre: 436209:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8027.800361] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8034.064067] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 8035.395712] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 8035.402686] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 8035.419073] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 8044.143397] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8061.912947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8061.995323] Lustre: 435192:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8062.063491] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8062.114667] Lustre: lustre-MDT0001: new disk, initializing [ 8062.171381] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8062.198239] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 8062.207398] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 8066.838139] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 8077.883351] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 8078.147343] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 8082.879064] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 8083.123186] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 8089.069575] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8089.075116] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 8089.094785] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8089.102048] Lustre: Skipped 1 previous similar message [ 8091.258605] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.21@tcp (stopping) [ 8093.905628] Lustre: server umount lustre-MDT0001 complete [ 8099.306167] LustreError: 435204:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8099.329964] LustreError: 435204:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 8104.417525] LustreError: 435203:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8104.437494] LustreError: 435203:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8109.540649] LustreError: 435200:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8109.582970] LustreError: 435200:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8114.657405] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8114.658261] LustreError: 435203:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8114.676955] Lustre: Skipped 1 previous similar message [ 8114.678547] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8118.453314] Lustre: server umount lustre-OST0000 complete [ 8119.783771] LustreError: 435204:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8129.188385] Lustre: server umount lustre-MDT0000 complete [ 8145.192593] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 8148.408833] Key type lgssc unregistered [ 8148.703287] LNet: 439959:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8148.710968] LNetError: 439959:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8148.758775] LNet: Removed LNI 192.168.202.121@tcp [ 8149.519784] Key type .llcrypt unregistered [ 8149.522966] Key type ._llcrypt unregistered [ 8194.102595] Key type ._llcrypt registered [ 8194.105432] Key type .llcrypt registered [ 8194.201784] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 8194.477488] Key type .llcrypt unregistered [ 8194.480233] Key type ._llcrypt unregistered [ 8212.517981] Key type ._llcrypt registered [ 8212.519980] Key type .llcrypt registered [ 8212.625704] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 8213.954152] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8214.072461] alg: No test for adler32 (adler32-zlib) [ 8215.219557] Lustre: Lustre: Build Version: 2.17.57_103_g794a134 [ 8215.461679] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 8217.208578] Key type lgssc registered [ 8218.608531] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8245.854218] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8245.868878] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8247.211122] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8247.257163] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8247.354115] Lustre: lustre-MDT0000: new disk, initializing [ 8247.458024] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8247.473943] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8252.304151] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 8263.370242] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8263.529347] Lustre: 443330:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8263.621281] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8263.627867] Lustre: Skipped 1 previous similar message [ 8263.733806] Lustre: lustre-MDT0001: new disk, initializing [ 8263.862787] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8263.923874] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8263.940151] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8268.605182] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 8278.909423] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8284.805399] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8291.069628] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8291.515301] Lustre: lustre-OST0000: new disk, initializing [ 8291.518963] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8291.525598] Lustre: 445300:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8291.674971] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8298.024364] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 8299.752235] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8299.768330] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8299.911906] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8308.554874] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8315.361269] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8315.363476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8315.375468] Lustre: Skipped 1 previous similar message [ 8315.400284] Lustre: Skipped 2 previous similar messages [ 8320.486385] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8320.495574] Lustre: Skipped 2 previous similar messages [ 8321.419939] Lustre: server umount lustre-MDT0000 complete [ 8325.292231] LustreError: 444126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788720554 with bad export cookie 17878766776712377197 [ 8325.293949] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8325.307498] LustreError: 444126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8325.679475] Lustre: server umount lustre-MDT0001 complete [ 8339.572356] Lustre: server umount lustre-OST0000 complete [ 8350.685989] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 14:49:39 (1788720579) [ 8360.146832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8360.776718] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8364.569653] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 8366.050090] LustreError: 447326:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8366.085749] LustreError: 447326:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8371.174860] LustreError: 447327:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8373.964138] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8374.342845] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8379.107098] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 8385.775996] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8391.083482] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8411.321597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8411.596165] Lustre: Found index 0 for lustre-OST0000, updating log [ 8411.797073] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8419.683388] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 8427.868975] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8432.316095] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 8437.737080] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8437.739626] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8437.768853] Lustre: Skipped 2 previous similar messages [ 8437.778141] Lustre: Skipped 1 previous similar message [ 8439.003655] Lustre: server umount lustre-OST0000 complete [ 8456.160978] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8456.163068] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8456.169542] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8456.172182] Lustre: Skipped 2 previous similar messages [ 8460.266106] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8460.269119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8460.309273] Lustre: Skipped 1 previous similar message [ 8461.866299] Lustre: server umount lustre-MDT0000 complete [ 8465.377310] LustreError: 447322:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8465.392086] LustreError: 447322:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8465.868201] LustreError: 447309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788720695 with bad export cookie 17878766776712378583 [ 8465.869988] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8465.878264] LustreError: 447309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8466.253968] Lustre: server umount lustre-MDT0001 complete [ 8475.894821] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 14:51:44 (1788720704) [ 8479.711936] LNet: 1 local NIs in recovery (showing 1): 192.168.202.121@tcp [ 9098.102517] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9098.457186] LustreError: 453749:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9098.552851] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9102.888957] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 9103.850580] LustreError: 453750:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9108.969397] LustreError: 453749:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9112.512969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9112.809640] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9118.598754] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 9126.663710] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9133.346976] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9153.212972] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9153.493494] Lustre: Found index 0 for lustre-OST0000, updating log [ 9153.775332] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9156.576909] LNet: 1 local NIs in recovery (showing 1): 192.168.202.121@tcp [ 9161.081990] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 9169.163489] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9172.514781] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9172.528807] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9177.996761] Lustre: server umount lustre-OST0000 complete [ 9182.822274] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9182.831835] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9182.839432] Lustre: Skipped 1 previous similar message [ 9182.853684] Lustre: Skipped 1 previous similar message [ 9184.929763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9184.944846] Lustre: Skipped 3 previous similar messages [ 9188.224229] Lustre: server umount lustre-MDT0000 complete [ 9189.090835] LustreError: 454477:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9189.131345] LustreError: 454477:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9192.849423] LustreError: 456554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788721422 with bad export cookie 17878766776712379668 [ 9192.857397] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9192.876151] LustreError: 456554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9193.384929] Lustre: server umount lustre-MDT0001 complete [ 9216.991500] LNet: 1 local NIs in recovery (showing 1): 192.168.202.121@tcp [ 9228.325374] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_hostid [ 9241.572883] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 9277.602909] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 15:05:06 (1788721506) [ 9300.738425] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 9304.007544] Key type lgssc unregistered [ 9304.305655] LNet: 460369:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9304.324882] LNetError: 460369:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9413.932167] LNet: Removed LNI 192.168.202.121@tcp [ 9414.722976] Key type .llcrypt unregistered [ 9414.724200] Key type ._llcrypt unregistered [ 9431.750509] Key type ._llcrypt registered [ 9431.751649] Key type .llcrypt registered [ 9431.904522] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 9432.933915] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9433.088968] alg: No test for adler32 (adler32-zlib) [ 9434.180959] Lustre: Lustre: Build Version: 2.17.57_103_g794a134 [ 9434.472538] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 9436.151904] Key type lgssc registered [ 9437.464607] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9442.400199] LNet: Added LNI 192.168.202.121@tcp100 [8/256/0/180] [ 9443.210485] LNet: Added LNI 192.168.202.121@tcp101 [8/256/0/180] [ 9443.971389] LNet: Added LNI 192.168.202.121@tcp102 [8/256/0/180] [ 9444.815963] LNet: Added LNI 192.168.202.121@tcp103 [8/256/0/180] [ 9445.890516] LNet: Added LNI 192.168.202.121@tcp104 [8/256/0/180] [ 9446.802194] LNet: Added LNI 192.168.202.121@tcp105 [8/256/0/180] [ 9447.807771] LNet: Added LNI 192.168.202.121@tcp106 [8/256/0/180] [ 9448.911634] LNet: Added LNI 192.168.202.121@tcp107 [8/256/0/180] [ 9449.883833] LNet: Added LNI 192.168.202.121@tcp108 [8/256/0/180] [ 9450.816352] LNet: Added LNI 192.168.202.121@tcp109 [8/256/0/180] [ 9451.895830] LNet: Added LNI 192.168.202.121@tcp110 [8/256/0/180] [ 9453.099827] LNet: Added LNI 192.168.202.121@tcp111 [8/256/0/180] [ 9454.036402] LNet: Added LNI 192.168.202.121@tcp112 [8/256/0/180] [ 9454.982277] LNet: Added LNI 192.168.202.121@tcp113 [8/256/0/180] [ 9456.000472] LNet: Added LNI 192.168.202.121@tcp114 [8/256/0/180] [ 9456.869311] LNet: Added LNI 192.168.202.121@tcp115 [8/256/0/180] [ 9457.805272] LNet: Added LNI 192.168.202.121@tcp116 [8/256/0/180] [ 9458.699773] LNet: Added LNI 192.168.202.121@tcp117 [8/256/0/180] [ 9459.612379] LNet: Added LNI 192.168.202.121@tcp118 [8/256/0/180] [ 9460.437555] LNet: Added LNI 192.168.202.121@tcp119 [8/256/0/180] [ 9461.405952] LNet: Added LNI 192.168.202.121@tcp120 [8/256/0/180] [ 9462.386779] LNet: Added LNI 192.168.202.121@tcp121 [8/256/0/180] [ 9463.288264] LNet: Added LNI 192.168.202.121@tcp122 [8/256/0/180] [ 9464.198180] LNet: Added LNI 192.168.202.121@tcp123 [8/256/0/180] [ 9464.980463] LNet: Added LNI 192.168.202.121@tcp124 [8/256/0/180] [ 9465.969802] LNet: Added LNI 192.168.202.121@tcp125 [8/256/0/180] [ 9466.977588] LNet: Added LNI 192.168.202.121@tcp126 [8/256/0/180] [ 9467.900654] LNet: Added LNI 192.168.202.121@tcp127 [8/256/0/180] [ 9468.760851] LNet: Added LNI 192.168.202.121@tcp128 [8/256/0/180] [ 9469.713828] LNet: Added LNI 192.168.202.121@tcp129 [8/256/0/180] [ 9470.423033] LNet: Added LNI 192.168.202.121@tcp130 [8/256/0/180] [ 9471.212386] LNet: Added LNI 192.168.202.121@tcp131 [8/256/0/180] [ 9471.866303] LNet: Added LNI 192.168.202.121@tcp132 [8/256/0/180] [ 9472.781194] LNet: Added LNI 192.168.202.121@tcp133 [8/256/0/180] [ 9473.950231] LNet: Added LNI 192.168.202.121@tcp134 [8/256/0/180] [ 9474.877874] LNet: Added LNI 192.168.202.121@tcp135 [8/256/0/180] [ 9482.364068] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9482.393897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9483.553024] Lustre: 463394:0:(lustre_peer.c:163:class_add_uuid()) 192.168.202.121@tcp: can't add NID 192.168.202.121@tcp130: rc = -75 [ 9483.624120] LNetError: 463414:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.121 [ 9483.635465] LNetError: Protocol error connecting to 192.168.202.121@tcp115 at host 192.168.202.121:988: is it running a compatible version of Lustre? [ 9483.647979] LNetError: Skipped 4 previous similar messages [ 9483.679156] LNet: 8 local NIs in recovery (showing 8): 192.168.202.121@tcp119, 192.168.202.121@tcp122, 192.168.202.121@tcp123, 192.168.202.121@tcp125, 192.168.202.121@tcp126, 192.168.202.121@tcp128, 192.168.202.121@tcp127, 192.168.202.121@tcp129 [ 9483.754174] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9483.773324] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9483.862481] Lustre: lustre-MDT0000: new disk, initializing [ 9483.934563] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9483.947224] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9484.704926] LNetError: 463415:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.121 [ 9484.732853] LNetError: 463415:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 9 previous similar messages [ 9484.743811] LNetError: Protocol error connecting to 192.168.202.121@tcp118 at host 192.168.202.121:988: is it running a compatible version of Lustre? [ 9484.765432] LNetError: Skipped 8 previous similar messages [ 9486.756872] LNetError: 463415:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.121 [ 9486.770800] LNetError: 463415:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 7 previous similar messages [ 9486.775864] LNetError: Protocol error connecting to 192.168.202.121@tcp126 at host 192.168.202.121:988: is it running a compatible version of Lustre? [ 9486.793888] LNetError: Skipped 4 previous similar messages [ 9489.159326] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 9490.849323] LNetError: 463426:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.121 [ 9490.869758] LNetError: 463426:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 1 previous similar message [ 9490.898414] LNetError: Protocol error connecting to 192.168.202.121@tcp126 at host 192.168.202.121:988: is it running a compatible version of Lustre? [ 9498.021256] LNetError: 460852:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.121 [ 9498.030286] LNetError: Protocol error connecting to 192.168.202.121@tcp126 at host 192.168.202.121:988: is it running a compatible version of Lustre? [ 9500.113589] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9500.527434] Lustre: 463410:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9500.628828] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9500.637358] Lustre: Skipped 1 previous similar message [ 9500.776816] Lustre: lustre-MDT0001: new disk, initializing [ 9500.833107] Lustre: 464211:0:(lustre_peer.c:163:class_add_uuid()) 192.168.202.121@tcp: can't add NID 192.168.202.121@tcp130: rc = -75 [ 9500.841138] Lustre: 464211:0:(lustre_peer.c:163:class_add_uuid()) Skipped 11 previous similar messages [ 9500.996280] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9501.071485] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9501.103024] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9505.807035] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 9509.538019] Lustre: 464575:0:(lustre_peer.c:163:class_add_uuid()) 192.168.202.121@tcp: can't add NID 192.168.202.121@tcp130: rc = -75 [ 9509.570058] Lustre: 464575:0:(lustre_peer.c:163:class_add_uuid()) Skipped 11 previous similar messages [ 9513.376547] LNetError: 463414:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.121 [ 9513.389501] LNetError: Protocol error connecting to 192.168.202.121@tcp115 at host 192.168.202.121:988: is it running a compatible version of Lustre? [ 9518.300245] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9525.979533] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9529.761367] LNetError: 460853:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.121 [ 9529.780520] LNetError: 460853:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 9 previous similar messages [ 9529.793213] LNetError: Protocol error connecting to 192.168.202.121@tcp121 at host 192.168.202.121:988: is it running a compatible version of Lustre? [ 9529.804558] LNetError: Skipped 9 previous similar messages [ 9532.394154] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9532.859570] Lustre: lustre-OST0000: new disk, initializing [ 9532.865339] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9532.883688] Lustre: 465385:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9533.067141] Lustre: 465390:0:(lustre_peer.c:163:class_add_uuid()) 192.168.202.121@tcp: can't add NID 192.168.202.121@tcp130: rc = -75 [ 9533.080382] Lustre: 465390:0:(lustre_peer.c:163:class_add_uuid()) Skipped 5 previous similar messages [ 9533.145441] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9539.660466] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 9539.686778] Lustre: 465739:0:(lustre_peer.c:163:class_add_uuid()) 192.168.202.121@tcp: can't add NID 192.168.202.121@tcp130: rc = -75 [ 9539.695036] Lustre: 465739:0:(lustre_peer.c:163:class_add_uuid()) Skipped 11 previous similar messages [ 9539.755840] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9539.762237] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9539.929049] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9544.095219] LNet: 1 local NIs in recovery (showing 1): 192.168.202.121@tcp126 [ 9548.672065] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9560.548402] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9560.568024] Lustre: Skipped 1 previous similar message [ 9560.569635] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9560.591030] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9562.546610] Lustre: server umount lustre-OST0000 complete [ 9562.595090] LNetError: 463414:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.121 [ 9562.614959] LNetError: 463414:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 16 previous similar messages [ 9562.626284] LNetError: Protocol error connecting to 192.168.202.121@tcp125 at host 192.168.202.121:988: is it running a compatible version of Lustre? [ 9562.637927] LNetError: Skipped 16 previous similar messages [ 9567.712793] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9567.720975] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9567.748747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9567.752228] Lustre: Skipped 1 previous similar message [ 9570.784634] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9570.785352] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9572.058188] Lustre: server umount lustre-MDT0000 complete [ 9575.325888] LustreError: 463404:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788721804 with bad export cookie 934618470829751353 [ 9575.331776] LustreError: 463404:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9575.332319] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9575.508780] Lustre: server umount lustre-MDT0001 complete [ 9586.039350] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 9589.913589] Key type lgssc unregistered [ 9590.416818] LNet: 467297:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9590.425502] LNetError: 467297:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9590.442815] LNet: Removed LNI 192.168.202.121@tcp [ 9591.455956] LNet: Removed LNI 192.168.202.121@tcp100 [ 9591.461803] LNetError: 467297:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9591.482040] LNetError: 467297:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 9591.507143] LNet: Removed LNI 192.168.202.121@tcp101 [ 9592.543165] LNet: Removed LNI 192.168.202.121@tcp102 [ 9592.550657] LNetError: 467297:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9592.571428] LNetError: 467297:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 9592.602691] LNet: Removed LNI 192.168.202.121@tcp103 [ 9593.695164] LNet: Removed LNI 192.168.202.121@tcp104 [ 9594.719717] LNet: Removed LNI 192.168.202.121@tcp105 [ 9594.729976] LNetError: 467297:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9594.738050] LNetError: 467297:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 9594.742787] LNet: Removed LNI 192.168.202.121@tcp106 [ 9595.807161] LNet: Removed LNI 192.168.202.121@tcp107 [ 9596.831206] LNet: Removed LNI 192.168.202.121@tcp108 [ 9597.855207] LNet: Removed LNI 192.168.202.121@tcp109 [ 9598.880673] LNet: Removed LNI 192.168.202.121@tcp110 [ 9598.896702] LNetError: 467297:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9598.919340] LNetError: 467297:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 9598.922638] LNet: Removed LNI 192.168.202.121@tcp111 [ 9599.967931] LNet: Removed LNI 192.168.202.121@tcp112 [ 9600.998292] LNet: Removed LNI 192.168.202.121@tcp113 [ 9602.015719] LNet: Removed LNI 192.168.202.121@tcp114 [ 9603.039753] LNet: Removed LNI 192.168.202.121@tcp115 [ 9604.066356] LNet: Removed LNI 192.168.202.121@tcp116 [ 9605.087530] LNet: Removed LNI 192.168.202.121@tcp117 [ 9606.111522] LNet: Removed LNI 192.168.202.121@tcp118 [ 9607.135319] LNet: Removed LNI 192.168.202.121@tcp119 [ 9607.141313] LNetError: 467297:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9607.153533] LNetError: 467297:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [ 9607.167858] LNet: Removed LNI 192.168.202.121@tcp120 [ 9608.223202] LNet: Removed LNI 192.168.202.121@tcp121 [ 9609.247131] LNet: Removed LNI 192.168.202.121@tcp122 [ 9610.271230] LNet: Removed LNI 192.168.202.121@tcp123 [ 9611.295230] LNet: Removed LNI 192.168.202.121@tcp124 [ 9612.319331] LNet: Removed LNI 192.168.202.121@tcp125 [ 9612.331048] LNet: Removed LNI 192.168.202.121@tcp126 [ 9613.343400] LNet: Removed LNI 192.168.202.121@tcp127 [ 9614.367396] LNet: Removed LNI 192.168.202.121@tcp128 [ 9615.391179] LNet: Removed LNI 192.168.202.121@tcp129 [ 9615.394534] LNet: Removed LNI 192.168.202.121@tcp130 [ 9615.399792] LNet: Removed LNI 192.168.202.121@tcp131 [ 9615.402723] LNet: Removed LNI 192.168.202.121@tcp132 [ 9615.406732] LNet: Removed LNI 192.168.202.121@tcp133 [ 9615.410402] LNet: Removed LNI 192.168.202.121@tcp134 [ 9615.421835] LNet: Removed LNI 192.168.202.121@tcp135 [ 9616.272343] Key type .llcrypt unregistered [ 9616.275126] Key type ._llcrypt unregistered [ 9639.270980] Key type ._llcrypt registered [ 9639.274354] Key type .llcrypt registered [ 9639.411895] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_hostid [ 9656.830775] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 9659.193482] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9659.291406] alg: No test for adler32 (adler32-zlib) [ 9660.498543] Lustre: Lustre: Build Version: 2.17.57_103_g794a134 [ 9660.726743] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 9662.335747] Key type lgssc registered [ 9664.081479] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9702.046165] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 15:12:11 (1788721931) [ 9723.009897] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 9726.216346] Key type lgssc unregistered [ 9726.543698] LNet: 471384:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9726.551420] LNetError: 471384:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9726.577949] LNet: Removed LNI 192.168.202.121@tcp [ 9727.407421] Key type .llcrypt unregistered [ 9727.409303] Key type ._llcrypt unregistered [ 9744.149093] Key type ._llcrypt registered [ 9744.154878] Key type .llcrypt registered [ 9744.299133] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 9745.669507] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9745.832789] alg: No test for adler32 (adler32-zlib) [ 9746.883722] Lustre: Lustre: Build Version: 2.17.57_103_g794a134 [ 9747.078638] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 9748.880596] Key type lgssc registered [ 9750.770867] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9764.538231] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9764.562048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9765.869306] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9765.936114] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9766.028896] Lustre: lustre-MDT0000: new disk, initializing [ 9766.083485] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9766.103082] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9771.100605] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 9781.640656] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9781.740960] Lustre: 472596:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9781.814430] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9781.816788] Lustre: Skipped 1 previous similar message [ 9781.863236] Lustre: lustre-MDT0001: new disk, initializing [ 9781.914917] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9781.937870] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9781.943293] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9786.805180] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 9796.460641] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9803.693601] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9811.582658] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9811.973705] Lustre: lustre-OST0000: new disk, initializing [ 9811.976847] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9811.981992] Lustre: 474565:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9812.202066] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9819.240776] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9819.260739] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9819.395732] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9819.480624] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 9829.903378] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9834.380354] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing lnet_if_list [ 9840.330544] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [ 9840.359672] LNet: Added LNI 192.168.202.121@tcp10 [8/256/0/180] [ 9847.629838] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [ 9847.671247] LNet: Added LNI 192.168.202.121@tcp11 [8/256/0/180] [ 9853.494317] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [ 9853.541051] LNet: Added LNI 192.168.202.121@tcp12 [8/256/0/180] [ 9858.924494] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [ 9858.946622] LNet: Added LNI 192.168.202.121@tcp13 [8/256/0/180] [ 9864.352871] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [ 9864.380192] LNet: Added LNI 192.168.202.121@tcp14 [8/256/0/180] [ 9869.329422] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [ 9869.393716] LNet: Added LNI 192.168.202.121@tcp15 [8/256/0/180] [ 9873.829933] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [ 9873.866484] LNet: Added LNI 192.168.202.121@tcp16 [8/256/0/180] [ 9880.046753] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [ 9880.073949] LNet: Added LNI 192.168.202.121@tcp17 [8/256/0/180] [ 9886.988435] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [ 9887.030329] LNet: Added LNI 192.168.202.121@tcp18 [8/256/0/180] [ 9894.837389] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [ 9894.883698] LNet: Added LNI 192.168.202.121@tcp19 [8/256/0/180] [ 9902.365125] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [ 9902.403561] LNet: Added LNI 192.168.202.121@tcp20 [8/256/0/180] [ 9909.604292] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [ 9909.637915] LNet: Added LNI 192.168.202.121@tcp21 [8/256/0/180] [ 9915.886369] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [ 9915.923755] LNet: Added LNI 192.168.202.121@tcp22 [8/256/0/180] [ 9922.479983] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [ 9922.537842] LNet: Added LNI 192.168.202.121@tcp23 [8/256/0/180] [ 9929.074775] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [ 9929.118804] LNet: Added LNI 192.168.202.121@tcp24 [8/256/0/180] [ 9935.973395] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [ 9935.995392] LNet: Added LNI 192.168.202.121@tcp25 [8/256/0/180] [ 9941.092922] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [ 9941.114683] LNet: Added LNI 192.168.202.121@tcp26 [8/256/0/180] [ 9947.983383] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [ 9948.046433] LNet: Added LNI 192.168.202.121@tcp27 [8/256/0/180] [ 9955.473410] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [ 9955.506886] LNet: Added LNI 192.168.202.121@tcp28 [8/256/0/180] [ 9962.804076] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [ 9962.833182] LNet: Added LNI 192.168.202.121@tcp29 [8/256/0/180] [ 9970.563532] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [ 9970.580874] LNet: Added LNI 192.168.202.121@tcp30 [8/256/0/180] [ 9977.375267] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [ 9977.405314] LNet: Added LNI 192.168.202.121@tcp31 [8/256/0/180] [ 9983.606691] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [ 9983.623604] LNet: Added LNI 192.168.202.121@tcp32 [8/256/0/180] [ 9994.489457] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [ 9994.517338] LNet: Added LNI 192.168.202.121@tcp33 [8/256/0/180] [10004.490347] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [10004.523797] LNet: Added LNI 192.168.202.121@tcp34 [8/256/0/180] [10012.799772] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [10012.849337] LNet: Added LNI 192.168.202.121@tcp35 [8/256/0/180] [10021.034678] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [10021.069149] LNet: Added LNI 192.168.202.121@tcp36 [8/256/0/180] [10030.555352] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [10030.586980] LNet: Added LNI 192.168.202.121@tcp37 [8/256/0/180] [10039.089926] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [10039.140721] LNet: Added LNI 192.168.202.121@tcp38 [8/256/0/180] [10046.567989] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [10046.593333] LNet: Added LNI 192.168.202.121@tcp39 [8/256/0/180] [10054.676031] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [10054.709399] LNet: Added LNI 192.168.202.121@tcp40 [8/256/0/180] [10063.739699] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [10063.801653] LNet: Added LNI 192.168.202.121@tcp41 [8/256/0/180] [10071.800424] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [10071.818499] LNet: Added LNI 192.168.202.121@tcp42 [8/256/0/180] [10081.338957] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [10081.375118] LNet: Added LNI 192.168.202.121@tcp43 [8/256/0/180] [10091.531339] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [10091.553826] LNet: Added LNI 192.168.202.121@tcp44 [8/256/0/180] [10100.893753] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [10100.921658] LNet: Added LNI 192.168.202.121@tcp45 [8/256/0/180] [10112.165725] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [10112.188165] LNet: Added LNI 192.168.202.121@tcp46 [8/256/0/180] [10121.071256] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [10121.109510] LNet: Added LNI 192.168.202.121@tcp47 [8/256/0/180] [10130.209900] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [10130.235283] LNet: Added LNI 192.168.202.121@tcp48 [8/256/0/180] [10142.081983] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [10142.103374] LNet: Added LNI 192.168.202.121@tcp49 [8/256/0/180] [10152.097633] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [10152.171888] LNet: Added LNI 192.168.202.121@tcp50 [8/256/0/180] [10176.484258] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10176.492944] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10176.515351] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10178.531501] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10178.540565] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10178.573666] Lustre: Skipped 1 previous similar message [10181.117214] Lustre: server umount lustre-MDT0000 complete [10185.314988] LustreError: 474564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788722414 with bad export cookie 2775692792904223363 [10185.323593] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10185.333492] LustreError: 474564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10185.781842] Lustre: server umount lustre-MDT0001 complete [10205.157599] Lustre: 471865:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788722418/real 1788722418] req@000000007c7007f9 x1875610934896896/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788722434 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10205.206575] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10207.263248] Lustre: 471865:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788722420/real 1788722420] req@000000005063d78e x1875610934897024/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788722436 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10207.588572] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [10210.282934] Lustre: 471867:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788722423/real 1788722423] req@00000000722a79fa x1875610934897152/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788722439 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10216.415394] Lustre: 471866:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788722429/real 1788722429] req@00000000295be64d x1875610934897536/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788722445 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10216.438927] Lustre: 471866:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [10216.675691] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10220.575122] Lustre: 471866:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788722434/real 1788722434] req@00000000acd3e475 x1875610934897664/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788722450 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10230.231252] LustreError: 474557:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10230.258536] LustreError: 474557:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [10230.346289] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10233.451244] LustreError: 474558:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10233.470226] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10234.465820] LustreError: 474557:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10235.240431] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [10239.975664] LustreError: 474557:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10243.870079] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10244.321077] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10249.478894] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10249.526869] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [10258.015420] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10264.220920] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10285.539839] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10285.547170] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10285.559045] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10285.584115] Lustre: Skipped 1 previous similar message [10285.613158] Lustre: Skipped 3 previous similar messages [10290.663918] LustreError: 474546:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10290.676918] LustreError: 474546:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10290.808890] Lustre: server umount lustre-OST0000 complete [10295.776461] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10295.778788] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10295.791135] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10295.808620] Lustre: Skipped 1 previous similar message [10295.821662] Lustre: Skipped 1 previous similar message [10300.898332] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10300.905582] Lustre: Skipped 1 previous similar message [10301.780851] Lustre: server umount lustre-MDT0000 complete [10305.888975] LustreError: 473391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788722535 with bad export cookie 2775692792904226086 [10305.902675] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10305.908070] LustreError: 473391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10306.346585] Lustre: server umount lustre-MDT0001 complete [10317.344395] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [10320.145611] Key type lgssc unregistered [10320.521514] LNet: 501616:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10320.536390] LNetError: 501616:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10320.549098] LNet: Removed LNI 192.168.202.121@tcp [10320.550892] LNet: Removed LNI 192.168.202.121@tcp10 [10320.552244] LNet: Removed LNI 192.168.202.121@tcp11 [10320.553602] LNet: Removed LNI 192.168.202.121@tcp12 [10320.556926] LNet: Removed LNI 192.168.202.121@tcp13 [10320.559940] LNet: Removed LNI 192.168.202.121@tcp14 [10320.561618] LNet: Removed LNI 192.168.202.121@tcp15 [10320.562882] LNet: Removed LNI 192.168.202.121@tcp16 [10320.564966] LNet: Removed LNI 192.168.202.121@tcp17 [10320.566313] LNet: Removed LNI 192.168.202.121@tcp18 [10320.567877] LNet: Removed LNI 192.168.202.121@tcp19 [10320.570075] LNet: Removed LNI 192.168.202.121@tcp20 [10320.572376] LNet: Removed LNI 192.168.202.121@tcp21 [10320.574368] LNet: Removed LNI 192.168.202.121@tcp22 [10320.576960] LNet: Removed LNI 192.168.202.121@tcp23 [10320.578886] LNet: Removed LNI 192.168.202.121@tcp24 [10320.581238] LNet: Removed LNI 192.168.202.121@tcp25 [10320.587255] LNet: Removed LNI 192.168.202.121@tcp26 [10320.589581] LNet: Removed LNI 192.168.202.121@tcp27 [10320.591608] LNet: Removed LNI 192.168.202.121@tcp28 [10320.593662] LNet: Removed LNI 192.168.202.121@tcp29 [10320.595635] LNet: Removed LNI 192.168.202.121@tcp30 [10320.596916] LNet: Removed LNI 192.168.202.121@tcp31 [10320.598748] LNet: Removed LNI 192.168.202.121@tcp32 [10320.600454] LNet: Removed LNI 192.168.202.121@tcp33 [10320.602368] LNet: Removed LNI 192.168.202.121@tcp34 [10320.605352] LNet: Removed LNI 192.168.202.121@tcp35 [10320.607180] LNet: Removed LNI 192.168.202.121@tcp36 [10320.609789] LNet: Removed LNI 192.168.202.121@tcp37 [10320.611553] LNet: Removed LNI 192.168.202.121@tcp38 [10320.612995] LNet: Removed LNI 192.168.202.121@tcp39 [10320.614687] LNet: Removed LNI 192.168.202.121@tcp40 [10320.616524] LNet: Removed LNI 192.168.202.121@tcp41 [10320.617930] LNet: Removed LNI 192.168.202.121@tcp42 [10320.619262] LNet: Removed LNI 192.168.202.121@tcp43 [10320.620871] LNet: Removed LNI 192.168.202.121@tcp44 [10320.622472] LNet: Removed LNI 192.168.202.121@tcp45 [10320.624683] LNet: Removed LNI 192.168.202.121@tcp46 [10320.626140] LNet: Removed LNI 192.168.202.121@tcp47 [10320.627928] LNet: Removed LNI 192.168.202.121@tcp48 [10320.629936] LNet: Removed LNI 192.168.202.121@tcp49 [10320.637803] LNet: Removed LNI 192.168.202.121@tcp50 [10321.284166] Key type .llcrypt unregistered [10321.286752] Key type ._llcrypt unregistered [10341.589451] Key type ._llcrypt registered [10341.595811] Key type .llcrypt registered [10341.755593] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_hostid [10359.369754] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [10360.767574] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10360.817334] alg: No test for adler32 (adler32-zlib) [10361.841931] Lustre: Lustre: Build Version: 2.17.57_103_g794a134 [10362.066289] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [10363.746811] Key type lgssc registered [10364.921497] Lustre: Echo OBD driver; http://www.lustre.org/ [10401.559768] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10401.571416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10402.855076] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10402.906472] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10403.025360] Lustre: lustre-MDT0000: new disk, initializing [10403.189526] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10403.226191] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10407.836639] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [10419.228955] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10419.389574] Lustre: 504810:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10419.447964] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10419.452173] Lustre: Skipped 1 previous similar message [10419.524413] Lustre: lustre-MDT0001: new disk, initializing [10419.615441] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10419.650269] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10419.671855] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10424.632479] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [10434.229372] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10440.048541] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10448.401398] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10448.713953] Lustre: *** cfs_fail_loc=90f, val=0*** [10448.819345] Lustre: lustre-OST0000: new disk, initializing [10448.825831] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10448.838217] Lustre: 506826:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10448.948727] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10455.752471] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [10456.640197] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10456.654325] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10456.782686] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10464.791638] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10472.420313] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10472.423171] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10472.425672] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10472.441643] Lustre: Skipped 1 previous similar message [10472.463377] LustreError: Skipped 1 previous similar message [10477.538790] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10477.548290] Lustre: Skipped 1 previous similar message [10477.829763] Lustre: server umount lustre-OST0000 complete [10482.658576] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10482.665385] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10482.675723] Lustre: Skipped 1 previous similar message [10482.687357] Lustre: Skipped 1 previous similar message [10487.777052] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10487.786216] Lustre: Skipped 2 previous similar messages [10488.395251] Lustre: server umount lustre-MDT0000 complete [10491.860312] LustreError: 505604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788722721 with bad export cookie 15197038375998088250 [10491.864394] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10492.427714] Lustre: server umount lustre-MDT0001 complete [10503.751602] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [10507.333752] Key type lgssc unregistered [10507.866973] LNet: 508731:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10507.870797] LNetError: 508731:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10507.891990] LNet: Removed LNI 192.168.202.121@tcp [10509.116404] Key type .llcrypt unregistered [10509.122528] Key type ._llcrypt unregistered [10529.872434] Key type ._llcrypt registered [10529.883415] Key type .llcrypt registered [10530.121571] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [10531.514103] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10531.617945] alg: No test for adler32 (adler32-zlib) [10532.971495] Lustre: Lustre: Build Version: 2.17.57_103_g794a134 [10533.173673] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [10534.815156] Key type lgssc registered [10535.949876] Lustre: Echo OBD driver; http://www.lustre.org/ [10544.932204] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10544.961592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10546.525290] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10551.201423] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [10556.905945] LustreError: 510002:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10556.930067] LustreError: 510002:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10562.023367] LustreError: 510001:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10563.572668] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10564.146451] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10568.381652] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [10577.097818] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10583.807842] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10591.080049] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10591.483460] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10598.386343] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [10606.486858] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10619.004169] loop0: detected capacity change from 0 to 204800000 [10619.283945] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10619.305356] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10619.319697] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10626.753909] loop0: detected capacity change from 0 to 204800000 [10627.028661] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10627.042674] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10627.061585] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10632.705656] loop0: detected capacity change from 0 to 204800000 [10638.704619] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10638.886626] Lustre: fs15246-OST0000: new disk, initializing [10638.891640] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [10638.898078] Lustre: 513598:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10638.937078] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10645.363465] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [10655.757365] loop1: detected capacity change from 0 to 204800000 [10662.323144] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10662.517659] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [10662.602574] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [10662.756778] Lustre: fs15246-MDT0000: new disk, initializing [10662.999729] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10663.039424] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10663.193318] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10664.236846] Lustre: 514720:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000014c5d59b x1875611758936576/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788722909 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [10668.225239] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [10669.360226] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [10669.371037] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [10669.416292] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [10694.160835] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10694.187689] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [10697.293876] Lustre: server umount fs15246-MDT0000 complete [10699.241887] LustreError: 512756:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10704.363384] LustreError: 512756:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10707.507678] Lustre: server umount fs15246-OST0000 complete [10721.761830] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10721.774233] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10721.797079] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10724.841262] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10724.844345] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10727.325261] Lustre: server umount lustre-OST0000 complete [10733.024507] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10733.033208] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10733.055906] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10733.067142] Lustre: Skipped 1 previous similar message [10737.980731] Lustre: server umount lustre-MDT0000 complete [10740.205294] LustreError: 510730:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10740.235598] LustreError: 510730:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10741.854444] LustreError: 511844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788722971 with bad export cookie 3693434650510037519 [10741.865147] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10741.871324] LustreError: 511844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10742.291898] Lustre: server umount lustre-MDT0001 complete [10753.205244] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [10756.018366] Key type lgssc unregistered [10756.316783] LNet: 517817:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10756.323380] LNetError: 517817:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10756.353415] LNet: Removed LNI 192.168.202.121@tcp [10757.234579] Key type .llcrypt unregistered [10757.239575] Key type ._llcrypt unregistered [10779.105925] Key type ._llcrypt registered [10779.108439] Key type .llcrypt registered [10779.220173] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [10780.427929] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10780.527111] alg: No test for adler32 (adler32-zlib) [10781.553176] Lustre: Lustre: Build Version: 2.17.57_103_g794a134 [10781.730701] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [10783.351697] Key type lgssc registered [10784.534890] Lustre: Echo OBD driver; http://www.lustre.org/ [10827.159202] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_hostid [10837.249318] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [10872.897106] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10872.941696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10874.297254] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10874.365364] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10874.529799] Lustre: lustre-MDT0000: new disk, initializing [10874.742735] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10874.776615] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10880.314750] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [10892.036589] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10892.171640] Lustre: 522282:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10892.226374] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10892.230017] Lustre: Skipped 1 previous similar message [10892.297572] Lustre: lustre-MDT0001: new disk, initializing [10892.466900] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10892.559225] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10892.570818] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10896.292664] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [10906.757959] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10913.334882] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10922.204942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10922.566829] Lustre: lustre-OST0000: new disk, initializing [10922.570693] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10922.589664] Lustre: 524253:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10922.773559] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10930.656709] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [10931.341618] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10931.361480] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10931.512512] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10941.404844] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10947.021249] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10947.322883] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10952.934115] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10953.252428] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10957.283322] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10957.285072] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10957.287721] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10957.291329] LustreError: Skipped 1 previous similar message [10957.299826] Lustre: Skipped 1 previous similar message [10962.007871] Lustre: server umount lustre-OST0000 complete [10967.524473] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10967.546736] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10967.549606] Lustre: Skipped 1 previous similar message [10972.641069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10972.643791] Lustre: Skipped 1 previous similar message [10973.062477] Lustre: server umount lustre-MDT0000 complete [10977.504895] LustreError: 522274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788723207 with bad export cookie 17965176546257318145 [10977.508720] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10977.515211] LustreError: 522274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10977.964392] Lustre: server umount lustre-MDT0001 complete [11005.794777] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_hostid [11016.984694] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [11055.891846] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 10769 sec ========= 15:34:45 (1788723285) [11057.810855] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 15:34:46 (1788723286) === [11062.209417] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 15:34:51 (1788723291) === [11107.036087] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [11111.610905] Key type lgssc unregistered [11112.083304] LNet: 530897:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11112.089153] LNetError: 530897:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11112.108509] LNet: Removed LNI 192.168.202.121@tcp [11113.454049] Key type .llcrypt unregistered [11113.465648] Key type ._llcrypt unregistered