[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 440902598 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001008] APIC: Switch to symmetric I/O mode setup [ 0.002225] x2apic enabled [ 0.003006] Switched APIC routing to physical x2apic. [ 0.004012] kvm-guest: setup PV IPIs [ 0.007289] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008018] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009009] pid_max: default: 32768 minimum: 301 [ 0.010134] LSM: Security Framework initializing [ 0.011041] Yama: becoming mindful. [ 0.012044] SELinux: Initializing. [ 0.013054] *** VALIDATE selinux *** [ 0.020667] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024462] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025144] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026101] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027100] *** VALIDATE tmpfs *** [ 0.029433] *** VALIDATE proc *** [ 0.030198] *** VALIDATE cgroup *** [ 0.031006] *** VALIDATE cgroup2 *** [ 0.032258] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034071] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036025] Spectre V2 : User space: Vulnerable [ 0.037005] Speculative Store Bypass: Vulnerable [ 0.040849] debug: unmapping init [mem 0xffffffffb4c59000-0xffffffffb4c60fff] [ 0.043151] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044686] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045018] ... version: 2 [ 0.046008] ... bit width: 48 [ 0.047007] ... generic registers: 4 [ 0.048009] ... value mask: 0000ffffffffffff [ 0.049009] ... max period: 00007fffffffffff [ 0.050010] ... fixed-purpose events: 3 [ 0.051008] ... event mask: 000000070000000f [ 0.052299] rcu: Hierarchical SRCU implementation. [ 0.054437] smp: Bringing up secondary CPUs ... [ 0.055552] x86: Booting SMP configuration: [ 0.056016] .... node #0, CPUs: #1 #2 #3 [ 0.059510] smp: Brought up 1 node, 4 CPUs [ 0.061010] smpboot: Max logical packages: 1 [ 0.062009] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.134611] node 0 deferred pages initialised in 69ms [ 0.137193] devtmpfs: initialized [ 0.138219] x86/mm: Memory block size: 128MB [ 0.140571] gcov: version magic: 0x41383552 [ 0.142073] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.143074] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.144239] pinctrl core: initialized pinctrl subsystem [ 0.145123] [ 0.145726] ************************************************************* [ 0.146010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.147011] ** ** [ 0.148010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.149012] ** ** [ 0.150010] ** This means that this kernel is built to expose internal ** [ 0.151013] ** IOMMU data structures, which may compromise security on ** [ 0.152010] ** your system. ** [ 0.153010] ** ** [ 0.154009] ** If you see this message and you are not debugging the ** [ 0.155011] ** kernel, report this immediately to your vendor! ** [ 0.156011] ** ** [ 0.157010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.158010] ************************************************************* [ 0.159669] NET: Registered protocol family 16 [ 0.160438] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.161048] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.162038] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.163415] cpuidle: using governor menu [ 0.167072] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.170394] PCI: Using configuration type 1 for base access [ 0.172088] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.179082] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.181015] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.184056] cryptd: max_cpu_qlen set to 1000 [ 0.186124] ACPI: Added _OSI(Module Device) [ 0.188013] ACPI: Added _OSI(Processor Device) [ 0.190008] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.191010] ACPI: Added _OSI(Processor Aggregator Device) [ 0.197257] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.203256] ACPI: Interpreter enabled [ 0.204037] ACPI: PM: (supports S0 S3 S4 S5) [ 0.205006] ACPI: Using IOAPIC for interrupt routing [ 0.206100] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.209388] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.219604] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.222042] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.224016] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.227077] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.232147] acpiphp: Slot [2] registered [ 0.234123] acpiphp: Slot [5] registered [ 0.235128] acpiphp: Slot [6] registered [ 0.237105] acpiphp: Slot [7] registered [ 0.238120] acpiphp: Slot [8] registered [ 0.239088] acpiphp: Slot [9] registered [ 0.240076] acpiphp: Slot [10] registered [ 0.241099] acpiphp: Slot [3] registered [ 0.242054] acpiphp: Slot [4] registered [ 0.243086] acpiphp: Slot [11] registered [ 0.244077] acpiphp: Slot [12] registered [ 0.246070] acpiphp: Slot [13] registered [ 0.247051] acpiphp: Slot [14] registered [ 0.248090] acpiphp: Slot [15] registered [ 0.249051] acpiphp: Slot [16] registered [ 0.250000] acpiphp: Slot [17] registered [ 0.250000] acpiphp: Slot [18] registered [ 0.252076] acpiphp: Slot [19] registered [ 0.253223] acpiphp: Slot [20] registered [ 0.255072] acpiphp: Slot [21] registered [ 0.256074] acpiphp: Slot [22] registered [ 0.258053] acpiphp: Slot [23] registered [ 0.259064] acpiphp: Slot [24] registered [ 0.260108] acpiphp: Slot [25] registered [ 0.262073] acpiphp: Slot [26] registered [ 0.263068] acpiphp: Slot [27] registered [ 0.265109] acpiphp: Slot [28] registered [ 0.266081] acpiphp: Slot [29] registered [ 0.268055] acpiphp: Slot [30] registered [ 0.269070] acpiphp: Slot [31] registered [ 0.270066] PCI host bridge to bus 0000:00 [ 0.270976] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.273016] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.274018] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.277012] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.279015] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.281018] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.283153] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.285897] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.289213] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.298000] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.302070] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.304014] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.306010] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.309016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.311209] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.313553] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.315028] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.317581] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.322010] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.332013] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.337014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.341000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.345016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.350014] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.366013] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.373986] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.378011] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.384011] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.395015] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.407000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.412019] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.418012] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.432018] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.442025] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.447011] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.451018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.464016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.472932] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.478012] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.483012] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.495015] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.504199] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.511026] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.517013] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.529016] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.540132] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.542319] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.545264] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.547336] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.549137] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.552381] iommu: Default domain type: Passthrough [ 0.554406] SCSI subsystem initialized [ 0.557160] ACPI: bus type USB registered [ 0.558113] usbcore: registered new interface driver usbfs [ 0.560080] usbcore: registered new interface driver hub [ 0.562107] usbcore: registered new device driver usb [ 0.564151] pps_core: LinuxPPS API ver. 1 registered [ 0.566009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.569055] PTP clock support registered [ 0.570115] EDAC MC: Ver: 3.0.0 [ 0.572508] PCI: Using ACPI for IRQ routing [ 0.573400] NetLabel: Initializing [ 0.574010] NetLabel: domain hash size = 128 [ 0.575007] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.577076] NetLabel: unlabeled traffic allowed by default [ 0.579113] vgaarb: loaded [ 0.580210] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.582008] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.587339] clocksource: Switched to clocksource kvm-clock [ 0.679064] VFS: Disk quotas dquot_6.6.0 [ 0.680223] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.682669] *** VALIDATE ramfs *** [ 0.683799] *** VALIDATE hugetlbfs *** [ 0.685186] pnp: PnP ACPI init [ 0.687381] pnp: PnP ACPI: found 6 devices [ 0.702135] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.704749] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.706463] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.708108] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.709940] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.711839] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.714072] NET: Registered protocol family 2 [ 0.716196] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.720467] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.723378] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.727379] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.730808] TCP: Hash tables configured (established 65536 bind 65536) [ 0.733287] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.735684] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.737936] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.740379] NET: Registered protocol family 1 [ 0.742452] RPC: Registered named UNIX socket transport module. [ 0.744715] RPC: Registered udp transport module. [ 0.747028] RPC: Registered tcp transport module. [ 0.749180] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.752024] NET: Registered protocol family 44 [ 0.754204] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.756038] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.757762] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.759694] PCI: CLS 0 bytes, default 64 [ 0.761169] Unpacking initramfs... [ 2.032769] debug: unmapping init [mem 0xffff8a793cc54000-0xffff8a793ffbffff] [ 2.035846] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.038230] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.041237] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.557705] Initialise system trusted keyrings [ 2.559155] Key type blacklist registered [ 2.560627] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.567925] zbud: loaded [ 2.571218] *** VALIDATE nfs *** [ 2.572138] *** VALIDATE nfs4 *** [ 2.573439] pstore: using deflate compression [ 2.577602] Platform Keyring initialized [ 2.670859] NET: Registered protocol family 38 [ 2.672704] Key type asymmetric registered [ 2.674390] Asymmetric key parser 'x509' registered [ 2.676377] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.679384] io scheduler mq-deadline registered [ 2.681075] io scheduler kyber registered [ 2.682658] io scheduler bfq registered [ 2.684809] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.687960] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.690846] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.693374] ACPI: Power Button [PWRF] [ 2.697473] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.702484] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.716628] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.722680] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.756671] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.784958] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.811238] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.814803] Non-volatile memory driver v1.3 [ 2.816293] Linux agpgart interface v0.103 [ 2.841519] virtio_blk virtio1: [vda] 134640 512-byte logical blocks (68.9 MB/65.7 MiB) [ 2.843511] vda: detected capacity change from 0 to 68935680 [ 2.854834] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.856931] vdb: detected capacity change from 0 to 1073741824 [ 2.868834] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.870846] vdc: detected capacity change from 0 to 2621440000 [ 2.889770] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.892688] vdd: detected capacity change from 0 to 2621440000 [ 2.911868] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.914574] vde: detected capacity change from 0 to 4294967296 [ 2.929320] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.931698] vdf: detected capacity change from 0 to 4294967296 [ 2.937230] libphy: Fixed MDIO Bus: probed [ 2.944246] usbcore: registered new interface driver usbserial_generic [ 2.946618] usbserial: USB Serial support registered for generic [ 2.949132] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.952730] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.954574] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.957488] mousedev: PS/2 mouse device common for all mice [ 2.960765] rtc_cmos 00:05: RTC can wake from S4 [ 2.964767] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.965790] rtc_cmos 00:05: registered as rtc0 [ 2.973451] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.976969] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.979475] intel_pstate: CPU model not supported [ 2.983213] hid: raw HID events driver (C) Jiri Kosina [ 2.990131] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.990971] usbcore: registered new interface driver usbhid [ 2.990974] usbhid: USB HID core driver [ 2.991132] drop_monitor: Initializing network drop monitor service [ 2.991247] Initializing XFRM netlink socket [ 2.991587] NET: Registered protocol family 10 [ 2.992537] Segment Routing with IPv6 [ 3.005327] NET: Registered protocol family 17 [ 3.008165] mpls_gso: MPLS GSO support [ 3.015429] RAS: Correctable Errors collector initialized. [ 3.018806] AVX version of gcm_enc/dec engaged. [ 3.020178] AES CTR mode by8 optimization enabled [ 3.085146] sched_clock: Marking stable (3085124499, 0)->(3895388955, -810264456) [ 3.088297] registered taskstats version 1 [ 3.090274] Loading compiled-in X.509 certificates [ 3.092590] zswap: loaded using pool lzo/zbud [ 3.112643] Key type big_key registered [ 3.122276] Key type encrypted registered [ 3.123461] ima: No TPM chip found, activating TPM-bypass! [ 3.124802] ima: Allocated hash algorithm: sha1 [ 3.126324] ima: No architecture policies found [ 3.127949] evm: Initialising EVM extended attributes: [ 3.129139] evm: security.selinux [ 3.129836] evm: security.ima [ 3.130550] evm: security.capability [ 3.131491] evm: HMAC attrs: 0x1 [ 3.133394] rtc_cmos 00:05: setting system clock to 2026-04-30 18:18:51 UTC (1777573131) [ 3.138328] debug: unmapping init [mem 0xffffffffb5c03000-0xffffffffb5dfffff] [ 3.140765] debug: unmapping init [mem 0xffffffffb4982000-0xffffffffb4c58fff] [ 3.149092] Write protecting the kernel read-only data: 28672k [ 3.152586] debug: unmapping init [mem 0xffffffffb3003000-0xffffffffb31fffff] [ 3.154226] debug: unmapping init [mem 0xffffffffb3914000-0xffffffffb39fffff] [ 3.185253] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.193606] systemd[1]: Detected virtualization kvm. [ 3.195363] systemd[1]: Detected architecture x86-64. [ 3.196836] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.220746] systemd[1]: No hostname configured. [ 3.222152] systemd[1]: Set hostname to . [ 3.223776] random: systemd: uninitialized urandom read (16 bytes read) [ 3.225516] systemd[1]: Initializing machine ID from random generator. [ 3.261771] random: ln: uninitialized urandom read (6 bytes read) [ 3.330908] random: systemd: uninitialized urandom read (16 bytes read) [ 3.333704] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.338065] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.345599] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Swap. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local File Systems. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Apply Kernel Variables... Starting Create Volatile Files and Directories... Starting Journal Service... [ OK ] Reached target Timers. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.900681] device-mapper: uevent: version 1.0.3 [ 3.902506] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.578336] random: fast init done [ 4.600992] virtio_net virtio0 ens2: renamed from eth0 [ 4.625907] scsi host0: ata_piix [ 4.688346] scsi host1: ata_piix [ 4.689969] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.692527] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.466618] dracut-initqueue[591]: RTNETLINK answers: File exists [ 9.637901] random: crng init done [ 9.639685] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.134510] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.364934] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.625595] SELinux: Disabled at runtime. [ 11.680844] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.689471] systemd[1]: Detected virtualization kvm. [ 11.691282] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.191413] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.194668] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.199717] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.203471] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.206669] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.213708] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.217772] systemd[1]: Listening on initctl Compatibility Named Pipe. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on Process Core Dump Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice User and Session Slice. [ OK ] Listening on udev Control Socket. Mounting POSIX Message Queue File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-getty.slice. [ 12.295908] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting Huge Pages File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Kernel Debug File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target Slices. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Starting Apply Kernel Variables... [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 12.690059] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.107359] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.146516] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.248507] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.273714] EDAC sbridge: Ver: 1.1.2 [ 14.851668] Key type dns_resolver registered [ 15.158185] NFS: Registering the id_resolver key type [ 15.160186] Key type id_resolver registered [ 15.161739] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started RPC Bind. [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg238-server login: [ 40.324165] libcfs: loading out-of-tree module taints kernel. [ 40.339737] Key type ._llcrypt registered [ 40.341336] Key type .llcrypt registered [ 40.384408] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 48.179562] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 48.793205] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 48.799425] alg: No test for adler32 (adler32-zlib) [ 49.791789] Lustre: Lustre: Build Version: 2.17.52_126_g9629464 [ 50.100478] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 50.173172] Key type lgssc registered [ 52.335422] Lustre: Echo OBD driver; http://www.lustre.org/ [ 59.338588] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 77.500594] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 83.059817] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 83.072556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 84.230356] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 84.249786] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 84.311555] Lustre: lustre-MDT0000: new disk, initializing [ 84.361887] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 84.369242] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 86.514297] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 95.491562] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 95.535461] Lustre: 6482:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 95.550922] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 95.553699] Lustre: Skipped 1 previous similar message [ 95.607185] Lustre: lustre-MDT0001: new disk, initializing [ 95.648766] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 95.664109] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 95.675176] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 98.124435] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 101.665086] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 107.630877] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 107.782397] Lustre: lustre-OST0000: new disk, initializing [ 107.784174] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 107.815538] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 110.915676] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 111.302502] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 111.308365] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 111.351782] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 117.678937] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 117.759239] Lustre: lustre-OST0001: new disk, initializing [ 117.762711] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 117.803989] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 119.626507] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 119.629854] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 119.653756] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 120.685210] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 127.778585] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 130.260429] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 133.069215] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing check_logdir /tmp/testlogs/ [ 135.809222] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing yml_node [ 137.901801] Lustre: DEBUG MARKER: Client: 2.17.52.126 [ 139.286815] Lustre: DEBUG MARKER: MDS: 2.17.52.126 [ 140.562401] Lustre: DEBUG MARKER: OSS: 2.17.52.126 [ 141.419046] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Apr 30 14:21:08 EDT 2026 [ 151.214502] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 151.989288] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 165.858577] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 165.860541] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 165.868081] Lustre: Skipped 3 previous similar messages [ 165.874126] Lustre: Skipped 3 previous similar messages [ 170.590968] Lustre: server umount lustre-MDT0000 complete [ 171.918741] LustreError: 6474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777573300 with bad export cookie 11828061970211927977 [ 171.920700] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 171.922966] LustreError: 6474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 176.097297] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 176.097351] LustreError: 6489:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 176.102128] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 176.106231] LustreError: 6489:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 6 previous similar messages [ 176.107937] Lustre: Skipped 1 previous similar message [ 181.216868] LustreError: 11149:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 181.216933] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 181.224951] LustreError: 11149:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 181.227130] Lustre: Skipped 1 previous similar message [ 182.241189] LustreError: 6488:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 182.248163] LustreError: 6488:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 186.336276] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 186.418880] Lustre: server umount lustre-MDT0001 complete [ 188.007208] Lustre: server umount lustre-OST0000 complete [ 189.465257] Lustre: server umount lustre-OST0001 complete [ 191.657018] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 194.883449] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 208.565283] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 208.679103] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 208.696096] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 208.745664] Lustre: lustre-MDT0000: new disk, initializing [ 208.774547] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 208.782680] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 210.289663] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 214.845551] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 214.906605] Lustre: 14942:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 215.021506] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 216.545813] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 219.883060] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 221.112304] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 223.473922] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 223.575407] Lustre: lustre-OST0000: new disk, initializing [ 223.576803] Lustre: Skipped 1 previous similar message [ 223.578566] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 223.580703] Lustre: Skipped 2 previous similar messages [ 223.610844] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 223.613292] Lustre: Skipped 1 previous similar message [ 225.216691] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 225.220419] Lustre: Skipped 1 previous similar message [ 225.222876] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 225.263072] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 225.608238] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 228.921598] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 230.453711] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 230.541269] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 232.002879] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 232.081798] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 235.491283] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 235.491559] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 235.494240] LustreError: Skipped 1 previous similar message [ 235.499605] Lustre: Skipped 1 previous similar message [ 235.503066] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 235.506478] Lustre: Skipped 3 previous similar messages [ 240.611928] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 240.614750] Lustre: Skipped 1 previous similar message [ 247.264101] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 247.321449] Lustre: server umount lustre-OST0000 complete [ 250.849477] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 250.851570] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 250.854156] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 250.861293] Lustre: Skipped 1 previous similar message [ 250.863930] Lustre: Skipped 2 previous similar messages [ 263.136109] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 263.209789] Lustre: server umount lustre-MDT0000 complete [ 264.783656] LustreError: 14935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777573393 with bad export cookie 11828061970211929510 [ 264.788072] LustreError: 14935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 264.788135] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 264.930740] Lustre: server umount lustre-MDT0001 complete [ 266.518230] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 14:23:14 (1777573394) [ 267.425074] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 268.080946] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 14:23:15 (1777573395) [ 270.511550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 270.665479] LustreError: 18945:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 270.689926] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 272.051403] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 275.071815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 276.641050] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 278.653429] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 280.016617] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 282.453464] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 284.819962] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 286.970258] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 301.026317] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 301.029280] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 301.029530] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 301.029535] Lustre: Skipped 5 previous similar messages [ 301.032138] Lustre: Skipped 1 previous similar message [ 306.514415] Lustre: server umount lustre-OST0000 complete [ 311.267410] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 311.267436] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 311.275791] Lustre: Skipped 1 previous similar message [ 313.947756] Lustre: server umount lustre-MDT0000 complete [ 315.380473] LustreError: 18925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777573443 with bad export cookie 11828061970211930392 [ 315.383780] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 315.384229] LustreError: 18925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 315.509033] Lustre: server umount lustre-MDT0001 complete [ 319.774343] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 320.952658] Key type lgssc unregistered [ 321.114477] LNet: 22140:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 321.118761] LNetError: 22140:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 322.150379] LNet: Removed LNI 192.168.202.138@tcp [ 322.522151] Key type .llcrypt unregistered [ 322.523744] Key type ._llcrypt unregistered [ 331.718113] Key type ._llcrypt registered [ 331.719824] Key type .llcrypt registered [ 331.770712] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 332.165382] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 332.189128] alg: No test for adler32 (adler32-zlib) [ 333.051945] Lustre: Lustre: Build Version: 2.17.52_126_g9629464 [ 333.143540] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 333.199898] Key type lgssc registered [ 335.114706] Lustre: Echo OBD driver; http://www.lustre.org/ [ 338.421223] loop: module loaded [ 338.455270] loop0: detected capacity change from 0 to 204800000 [ 338.583705] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 338.588238] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 338.722510] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 341.089598] loop0: detected capacity change from 0 to 204800000 [ 341.201676] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 341.205922] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 341.210954] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 343.209333] loop0: detected capacity change from 0 to 204800000 [ 345.436095] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 345.443678] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 346.561674] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 346.587494] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 346.640802] Lustre: lustre-MDT0000: new disk, initializing [ 346.692173] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 346.699380] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 348.177356] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 352.336370] loop1: detected capacity change from 0 to 204800000 [ 354.369498] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 354.462472] Lustre: lustre-OST07c6: new disk, initializing [ 354.464218] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 354.466104] Lustre: Skipped 1 previous similar message [ 354.489490] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 356.643399] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 362.500775] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 362.504851] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 362.517415] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 365.299600] Lustre: 25278:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 365.303646] Lustre: 25278:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 365.307390] Lustre: 25278:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 365.310698] Lustre: 25278:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 365.314233] Lustre: 25278:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 365.317665] Lustre: 25278:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 379.682130] Lustre: 24274:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 379.685925] Lustre: 24274:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 379.688317] Lustre: 24274:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 379.692113] Lustre: 24274:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 379.695513] Lustre: 24274:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 379.698424] Lustre: 24274:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 383.311102] Lustre: 24274:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 383.315527] Lustre: 24274:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 383.319173] Lustre: 24274:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 383.321895] Lustre: 24274:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 383.325745] Lustre: 24274:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 383.329631] Lustre: 24274:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 383.333023] Lustre: 24274:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 383.337030] Lustre: 24274:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 383.340019] Lustre: 24274:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 383.343203] Lustre: 24274:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 383.346042] Lustre: 24274:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 383.348888] Lustre: 24274:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 386.426959] Lustre: 25278:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 386.431421] Lustre: 25278:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 386.435017] Lustre: 25278:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 386.437600] Lustre: 25278:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 386.441272] Lustre: 25278:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 386.445335] Lustre: 25278:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 386.449449] Lustre: 25278:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 386.453433] Lustre: 25278:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 386.459043] Lustre: 25278:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 386.462619] Lustre: 25278:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 386.467132] Lustre: 25278:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 386.472646] Lustre: 25278:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 393.201059] Lustre: 25278:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 393.205994] Lustre: 25278:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages [ 393.208671] Lustre: 25278:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 393.210926] Lustre: 25278:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 393.213537] Lustre: 25278:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 393.215896] Lustre: 25278:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 393.218502] Lustre: 25278:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 393.221361] Lustre: 25278:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 393.224236] Lustre: 25278:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 393.226688] Lustre: 25278:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 393.229964] Lustre: 25278:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 393.232871] Lustre: 25278:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 403.498444] Lustre: 24275:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 403.502394] Lustre: 24275:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 403.505028] Lustre: 24275:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 403.507740] Lustre: 24275:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 403.510363] Lustre: 24275:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 403.513033] Lustre: 24275:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 403.515724] Lustre: 24275:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 403.518746] Lustre: 24275:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 403.522191] Lustre: 24275:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 403.524962] Lustre: 24275:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 403.527869] Lustre: 24275:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 403.531099] Lustre: 24275:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 449.507841] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 449.509556] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 449.518399] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 454.626980] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 459.232129] Lustre: lustre-OST07c6 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 459.286711] Lustre: server umount lustre-OST07c6 complete [ 463.113836] Lustre: server umount lustre-MDT0000 complete [ 468.972960] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 470.121394] Key type lgssc unregistered [ 470.257628] LNet: 27802:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 470.260524] LNetError: 27802:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 470.271405] LNet: Removed LNI 192.168.202.138@tcp [ 470.599163] Key type .llcrypt unregistered [ 470.600286] Key type ._llcrypt unregistered [ 478.558382] Key type ._llcrypt registered [ 478.559594] Key type .llcrypt registered [ 478.600735] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 478.943935] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 478.992943] alg: No test for adler32 (adler32-zlib) [ 479.860394] Lustre: Lustre: Build Version: 2.17.52_126_g9629464 [ 479.954799] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 479.999630] Key type lgssc registered [ 481.931137] Lustre: Echo OBD driver; http://www.lustre.org/ [ 484.701465] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 484.709429] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 485.888219] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 487.348313] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 490.484994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 490.630726] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 492.183833] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 494.289837] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 495.670321] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 498.382744] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 498.555327] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 500.817485] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 502.804113] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 510.886817] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 521.186820] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 521.186902] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 521.190443] LustreError: Skipped 1 previous similar message [ 521.196633] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 521.199255] Lustre: Skipped 1 previous similar message [ 523.920536] Lustre: server umount lustre-OST0000 complete [ 526.305617] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 526.306059] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 526.309072] Lustre: Skipped 2 previous similar messages [ 527.389414] Lustre: server umount lustre-MDT0000 complete [ 528.813440] LustreError: 29750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777573657 with bad export cookie 13218386873821875057 [ 528.817268] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 528.818025] LustreError: 29750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 534.981732] Lustre: server umount lustre-MDT0001 complete [ 538.485078] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 539.523622] Key type lgssc unregistered [ 539.656354] LNet: 32432:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 539.659425] LNetError: 32432:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 539.668417] LNet: Removed LNI 192.168.202.138@tcp [ 539.967048] Key type .llcrypt unregistered [ 539.968586] Key type ._llcrypt unregistered [ 547.544854] Key type ._llcrypt registered [ 547.545840] Key type .llcrypt registered [ 547.584229] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 547.954862] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 548.005186] alg: No test for adler32 (adler32-zlib) [ 548.857931] Lustre: Lustre: Build Version: 2.17.52_126_g9629464 [ 548.949815] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 548.996442] Key type lgssc registered [ 550.924029] Lustre: Echo OBD driver; http://www.lustre.org/ [ 553.714794] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 553.721071] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 554.892411] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 556.288916] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 559.589289] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 559.736067] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 561.247291] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 563.364557] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 564.775931] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 567.199682] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 567.323709] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 569.271326] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 571.195527] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 580.649234] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 590.820914] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 590.821347] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 590.825523] LustreError: Skipped 1 previous similar message [ 590.826286] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 590.829802] Lustre: Skipped 1 previous similar message [ 590.836834] Lustre: Skipped 1 previous similar message [ 591.957972] Lustre: server umount lustre-OST0000 complete [ 595.937764] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 595.938015] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 595.945093] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 595.947241] Lustre: Skipped 1 previous similar message [ 599.515854] Lustre: server umount lustre-MDT0000 complete [ 600.977365] LustreError: 33643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777573729 with bad export cookie 14577608987136191392 [ 600.981399] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 600.982017] LustreError: 33643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 601.124590] Lustre: server umount lustre-MDT0001 complete [ 604.947254] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 606.049726] Key type lgssc unregistered [ 606.194401] LNet: 36856:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 606.197791] LNetError: 36856:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 606.209558] LNet: Removed LNI 192.168.202.138@tcp [ 606.516097] Key type .llcrypt unregistered [ 606.517366] Key type ._llcrypt unregistered [ 615.750950] Key type ._llcrypt registered [ 615.752381] Key type .llcrypt registered [ 615.791448] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 616.130306] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 616.177049] alg: No test for adler32 (adler32-zlib) [ 617.033191] Lustre: Lustre: Build Version: 2.17.52_126_g9629464 [ 617.117246] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 617.165877] Key type lgssc registered [ 619.109458] Lustre: Echo OBD driver; http://www.lustre.org/ [ 621.932718] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 621.941738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 623.108208] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 624.426977] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 627.379370] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 627.507109] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 628.851456] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 630.756975] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 632.186991] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 634.637688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 634.763303] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 636.678496] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 638.717861] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 649.129213] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 653.281741] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 653.287634] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 653.292791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 654.305495] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 654.308857] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 655.027366] Lustre: server umount lustre-MDT0000 complete [ 655.549298] LustreError: 39846:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 655.559090] LustreError: 39846:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 656.654327] LustreError: 38947:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777573785 with bad export cookie 10153992829209312165 [ 656.656584] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 656.659546] LustreError: 38947:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 659.425308] LustreError: 39541:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 659.425586] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 659.436489] Lustre: Skipped 1 previous similar message [ 659.438862] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 659.442067] Lustre: Skipped 2 previous similar messages [ 660.655143] LustreError: 38226:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 661.473196] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 661.475035] Lustre: Skipped 1 previous similar message [ 662.831909] Lustre: server umount lustre-MDT0001 complete [ 682.225592] Lustre: server umount lustre-OST0000 complete [ 694.579235] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 696.610263] Key type lgssc unregistered [ 696.803803] LNet: 41754:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 696.808392] LNetError: 41754:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 696.819925] LNet: Removed LNI 192.168.202.138@tcp [ 697.439163] Key type .llcrypt unregistered [ 697.440879] Key type ._llcrypt unregistered [ 719.796712] Key type ._llcrypt registered [ 719.799988] Key type .llcrypt registered [ 719.895347] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 721.036669] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 721.126949] alg: No test for adler32 (adler32-zlib) [ 722.164524] Lustre: Lustre: Build Version: 2.17.52_126_g9629464 [ 722.333489] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 722.431031] Key type lgssc registered [ 724.883863] Lustre: Echo OBD driver; http://www.lustre.org/ [ 729.889134] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 729.903174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 731.187509] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 733.374210] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 737.991365] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 738.227419] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 740.595914] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 743.705549] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 745.817910] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 749.428608] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 749.628516] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 752.826376] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 755.757257] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 755.971189] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 765.921695] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 765.924942] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 765.939218] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 767.971052] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 767.974985] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 767.975419] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 767.987616] Lustre: Skipped 1 previous similar message [ 770.958253] Lustre: server umount lustre-OST0000 complete [ 788.065745] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 788.069430] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 788.073934] Lustre: Skipped 1 previous similar message [ 788.081245] Lustre: Skipped 2 previous similar messages [ 792.188899] Lustre: server umount lustre-MDT0000 complete [ 793.569715] LustreError: 43155:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 793.582787] LustreError: 43155:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 794.044251] LustreError: 44768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777573922 with bad export cookie 10697559555809834630 [ 794.048314] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 794.050991] LustreError: 44768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 800.286806] Lustre: server umount lustre-MDT0001 complete [ 805.730399] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 807.428748] Key type lgssc unregistered [ 807.623424] LNet: 46504:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 807.628758] LNetError: 46504:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 807.642252] LNet: Removed LNI 192.168.202.138@tcp [ 808.074502] Key type .llcrypt unregistered [ 808.075617] Key type ._llcrypt unregistered [ 819.693863] Key type ._llcrypt registered [ 819.696589] Key type .llcrypt registered [ 819.764185] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 820.164359] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 820.230372] alg: No test for adler32 (adler32-zlib) [ 821.107216] Lustre: Lustre: Build Version: 2.17.52_126_g9629464 [ 821.221872] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 821.298632] Key type lgssc registered [ 823.396767] Lustre: Echo OBD driver; http://www.lustre.org/ [ 827.057077] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 827.066117] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 828.311334] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 830.616403] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 835.746803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 835.984841] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 838.286352] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 841.234940] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 843.156214] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 847.242772] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 847.478972] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 850.785309] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 853.620583] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 854.281651] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 862.759501] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 865.069675] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 876.563270] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-04-3014h33m23s [ 877.585943] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 881.638530] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 881.649031] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 881.652912] Lustre: Skipped 2 previous similar messages [ 881.666683] Lustre: Skipped 1 previous similar message [ 882.399220] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.38@tcp (stopping) [ 883.434425] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.38@tcp (stopping) [ 883.438535] Lustre: Skipped 3 previous similar messages [ 885.073070] Lustre: server umount lustre-MDT0000 complete [ 887.385408] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 888.491524] LustreError: 47876:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 888.504815] LustreError: 47876:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 890.905412] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 890.966801] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 891.106785] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 891.146068] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 892.140170] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 893.442399] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 895.357642] Lustre: DEBUG MARKER: Wait for df (35931) ... [ 896.332909] Lustre: DEBUG MARKER: done [ 901.601995] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 901.606053] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 901.622649] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 902.115360] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 904.616190] Lustre: server umount lustre-OST0000 complete [ 907.746181] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 907.748121] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 907.756058] Lustre: Skipped 2 previous similar messages [ 907.759519] Lustre: Skipped 2 previous similar messages [ 907.766234] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 912.612692] Lustre: server umount lustre-MDT0000 complete [ 913.827629] LustreError: 47882:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 913.839302] LustreError: 47882:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 4 previous similar messages [ 914.513561] LustreError: 48597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777574042 with bad export cookie 11957620253755902895 [ 914.516100] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 914.521067] LustreError: 48597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 914.685603] Lustre: server umount lustre-MDT0001 complete [ 920.119939] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 921.544608] Key type lgssc unregistered [ 921.732731] LNet: 52427:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 921.738293] LNetError: 52427:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 921.750704] LNet: Removed LNI 192.168.202.138@tcp [ 922.208095] Key type .llcrypt unregistered [ 922.210403] Key type ._llcrypt unregistered [ 942.277506] Key type ._llcrypt registered [ 942.279356] Key type .llcrypt registered [ 942.331601] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 942.811965] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 942.824833] alg: No test for adler32 (adler32-zlib) [ 943.712707] Lustre: Lustre: Build Version: 2.17.52_126_g9629464 [ 943.828077] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 943.902106] Key type lgssc registered [ 945.883087] Lustre: Echo OBD driver; http://www.lustre.org/ [ 949.200882] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 949.217956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 950.343417] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 950.356851] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 950.431769] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 952.089659] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 955.522337] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 955.569894] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 955.582612] Lustre: 54596:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 955.699089] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 957.180139] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 959.503418] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 961.085456] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 963.789228] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 963.861867] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 963.928732] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 964.226410] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 966.187867] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 968.138374] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 971.593205] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-04-3014h34m59s [ 972.145513] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 974.627805] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 975.551783] Lustre: DEBUG MARKER: Creating a test file and stat it [ 975.570493] LustreError: 54601:0:(mdt_handler.c:2347:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 996.024270] Lustre: lustre-MDT0000: Client 9805acd9-f6a0-4943-825f-7d6fa743f5ad (at 192.168.202.38@tcp) reconnecting [ 1015.616121] LustreError: 54601:0:(mdt_handler.c:2347:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1016.320549] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1017.135214] Lustre: DEBUG MARKER: done [ 1020.898041] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1020.898344] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1020.901375] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1020.908220] Lustre: Skipped 1 previous similar message [ 1024.861710] Lustre: server umount lustre-OST0000 complete [ 1027.552840] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1027.555330] LustreError: Skipped 1 previous similar message [ 1027.556937] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1027.561989] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1027.564490] Lustre: Skipped 1 previous similar message [ 1028.577437] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1028.579332] Lustre: Skipped 2 previous similar messages [ 1032.347688] Lustre: server umount lustre-MDT0000 complete [ 1033.639901] LustreError: 54588:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777574162 with bad export cookie 11804521560539746656 [ 1033.640312] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1033.644666] LustreError: 54588:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1033.769041] Lustre: server umount lustre-MDT0001 complete [ 1037.381766] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 1038.368536] Key type lgssc unregistered [ 1038.491797] LNet: 58562:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1038.495205] LNetError: 58562:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1038.501701] LNet: Removed LNI 192.168.202.138@tcp [ 1038.821913] Key type .llcrypt unregistered [ 1038.822949] Key type ._llcrypt unregistered [ 1055.360715] Key type ._llcrypt registered [ 1055.362156] Key type .llcrypt registered [ 1055.403530] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 1055.819358] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1055.834749] alg: No test for adler32 (adler32-zlib) [ 1056.698824] Lustre: Lustre: Build Version: 2.17.52_126_g9629464 [ 1056.790905] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 1056.840151] Key type lgssc registered [ 1058.776191] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1059.897434] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1059.904277] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1060.993096] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1061.000560] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1061.059089] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1061.413570] Lustre: Failing over lustre-MDT0000 [ 1061.529594] Lustre: server umount lustre-MDT0000 complete [ 1063.755697] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 14:36:31 (1777574191) [ 1066.113628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1066.252207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1067.658736] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 1071.187430] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1071.235898] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1071.251152] Lustre: 61319:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1071.365101] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1072.847602] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 1074.805123] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1076.178262] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1078.780884] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1078.854953] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1078.932518] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1081.219745] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 1082.523377] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1083.285189] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1086.384764] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1092.065057] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1092.069151] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1092.074080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1093.089368] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1093.090037] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1093.091774] Lustre: Skipped 1 previous similar message [ 1093.096544] Lustre: Skipped 1 previous similar message [ 1094.354345] Lustre: server umount lustre-MDT0000 complete [ 1095.575986] LustreError: 61309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777574223 with bad export cookie 13760523633195668873 [ 1095.578886] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1095.580413] LustreError: 61309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1095.687592] Lustre: server umount lustre-MDT0001 complete [ 1096.832801] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1098.415072] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1101.466598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1105.376680] LustreError: 60239:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000005345d5fb x1863921628641536/t0(0) o250->MGC192.168.202.138@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1105.550259] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1105.591148] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1107.111317] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 1109.977582] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1110.072325] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1111.428147] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 1113.187059] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1113.405785] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1114.208161] Lustre: 60241:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777574226/real 1777574226] req@000000002b48f925 x1863921628641152/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1777574242 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1114.691653] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1117.601866] Lustre: 60242:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777574228/real 1777574228] req@00000000d082aad7 x1863921628641280/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1777574244 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1118.691030] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1118.692236] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1118.697695] Lustre: Skipped 1 previous similar message [ 1119.712150] Lustre: 60243:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777574231/real 1777574231] req@0000000053332d89 x1863921628641408/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1777574247 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1122.725111] Lustre: server umount lustre-MDT0000 complete [ 1123.808866] LustreError: 65868:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1123.814211] LustreError: 65868:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 4 previous similar messages [ 1124.140242] LustreError: 61310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777574252 with bad export cookie 13760523633195671141 [ 1124.143625] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1124.144592] LustreError: 61310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1124.252382] Lustre: server umount lustre-MDT0001 complete [ 1126.657181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1130.528612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1134.048574] LustreError: 60239:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000001c1d1822 x1863921628660864/t0(0) o250->MGC192.168.202.138@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1134.156651] LustreError: 64766:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1134.183556] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1134.185974] Lustre: Skipped 1 previous similar message [ 1134.209355] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1135.471089] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 1138.427230] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1138.529180] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1138.534566] Lustre: Skipped 1 previous similar message [ 1139.988969] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 1141.993372] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1143.184743] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1143.781090] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1143.782821] Lustre: Skipped 1 previous similar message [ 1144.800183] Lustre: 60243:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777574257/real 1777574257] req@00000000e52bde88 x1863921628660480/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1777574273 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1144.807067] Lustre: 60243:0:(client.c:2479:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1148.901507] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1149.904126] Lustre: 60242:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777574262/real 1777574262] req@00000000795a28e1 x1863921628660992/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1777574278 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1149.911664] Lustre: 60242:0:(client.c:2479:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1154.017469] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1154.018322] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1154.020849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1154.023737] Lustre: Skipped 1 previous similar message [ 1154.027034] Lustre: Skipped 4 previous similar messages [ 1156.247755] Lustre: server umount lustre-MDT0000 complete [ 1157.509265] LustreError: 61310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777574285 with bad export cookie 13760523633195672520 [ 1157.511936] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1157.513670] LustreError: 61310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1157.610115] Lustre: server umount lustre-MDT0001 complete [ 1159.799966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1160.893049] Lustre: DEBUG MARKER: files compared the same [ 1170.781952] Lustre: server umount lustre-OST0000 complete [ 1175.968072] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 1177.011680] Key type lgssc unregistered [ 1177.134628] LNet: 70226:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1177.137465] LNetError: 70226:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1177.145591] LNet: Removed LNI 192.168.202.138@tcp [ 1177.424691] Key type .llcrypt unregistered [ 1177.425988] Key type ._llcrypt unregistered [ 1186.227077] Key type ._llcrypt registered [ 1186.228043] Key type .llcrypt registered [ 1186.265081] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 1186.641598] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1186.674636] alg: No test for adler32 (adler32-zlib) [ 1187.521895] Lustre: Lustre: Build Version: 2.17.52_126_g9629464 [ 1187.596624] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 1187.640964] Key type lgssc registered [ 1189.512503] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1191.910252] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1191.915336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1193.060057] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1194.278381] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug +malloc all [ 1197.047801] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1197.148255] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1198.434178] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug +malloc all [ 1200.134072] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1201.255728] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1203.401549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1203.484778] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1204.515911] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1205.227419] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug +malloc all [ 1206.831925] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1212.897380] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1212.897640] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1212.900419] Lustre: Skipped 1 previous similar message [ 1212.903161] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1216.271380] Lustre: server umount lustre-OST0000 complete [ 1217.504724] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1217.507608] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1217.512545] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1217.514821] Lustre: Skipped 1 previous similar message [ 1223.136582] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1223.138628] Lustre: Skipped 3 previous similar messages [ 1223.569362] Lustre: server umount lustre-MDT0000 complete [ 1224.751663] LustreError: 71582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777574353 with bad export cookie 3991458525440333253 [ 1224.752987] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1224.755195] LustreError: 71582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1224.849246] Lustre: server umount lustre-MDT0001 complete [ 1228.047858] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 1229.035746] Key type lgssc unregistered [ 1229.167330] LNet: 74774:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1229.170865] LNetError: 74774:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1229.180470] LNet: Removed LNI 192.168.202.138@tcp [ 1229.475660] Key type .llcrypt unregistered [ 1229.476789] Key type ._llcrypt unregistered [ 1238.163604] Key type ._llcrypt registered [ 1238.164566] Key type .llcrypt registered [ 1238.199178] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 1238.579333] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1238.606785] alg: No test for adler32 (adler32-zlib) [ 1239.458465] Lustre: Lustre: Build Version: 2.17.52_126_g9629464 [ 1239.538024] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 1239.581428] Key type lgssc registered [ 1241.472361] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1243.928497] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1243.935278] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1259.360296] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1264.480203] LustreError: 76125:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.138@tcp: failed processing log, type 4: rc = -110 [ 1291.104199] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1292.815036] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 1294.455843] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1296.989551] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1297.026835] Lustre: *** cfs_fail_loc=706, val=0*** [ 1297.148250] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1297.176335] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 1298.369981] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 1301.193511] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1301.313972] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1302.582800] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 1304.270112] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1305.398710] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1306.595045] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1306.595270] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1306.599056] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1306.601919] Lustre: Skipped 1 previous similar message [ 1306.606670] Lustre: Skipped 1 previous similar message [ 1308.109766] Lustre: server umount lustre-OST0000 complete [ 1311.714284] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1311.714750] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1311.717359] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1311.720422] Lustre: Skipped 1 previous similar message [ 1311.723429] Lustre: Skipped 1 previous similar message [ 1315.413148] Lustre: server umount lustre-MDT0000 complete [ 1316.578059] LustreError: 76131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777574444 with bad export cookie 17416636760330002727 [ 1316.580042] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1316.581391] LustreError: 76131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1316.689080] Lustre: server umount lustre-MDT0001 complete [ 1320.029923] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 1320.993310] Key type lgssc unregistered [ 1321.121352] LNet: 79385:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1321.123706] LNetError: 79385:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1321.135269] LNet: Removed LNI 192.168.202.138@tcp [ 1321.413160] Key type .llcrypt unregistered [ 1321.414201] Key type ._llcrypt unregistered [ 1330.395843] Key type ._llcrypt registered [ 1330.397066] Key type .llcrypt registered [ 1330.433651] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 1330.836359] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1330.842423] alg: No test for adler32 (adler32-zlib) [ 1331.688669] Lustre: Lustre: Build Version: 2.17.52_126_g9629464 [ 1331.758627] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 1331.800018] Key type lgssc registered [ 1333.641659] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1335.949147] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1335.955402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1338.252536] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 1340.844510] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1341.044756] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1342.429658] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 1345.305267] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1345.427222] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1347.373692] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 1350.071225] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1350.076331] Lustre: MGS: Client a463cfbc-7d81-4550-8a36-05c0464a2d32 (at 0@lo) reconnecting [ 1350.085179] LustreError: 81443:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 1350.164264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1350.186938] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 1351.425646] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 1360.353744] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1360.357406] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1360.362983] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1361.378836] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1361.379046] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1361.379302] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1365.199410] Lustre: server umount lustre-OST0000 complete [ 1366.497205] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1366.497509] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1366.504937] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1366.507835] Lustre: Skipped 2 previous similar messages [ 1371.617234] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1371.619281] Lustre: Skipped 1 previous similar message [ 1372.616313] Lustre: server umount lustre-MDT0000 complete [ 1373.970896] Lustre: server umount lustre-MDT0001 complete [ 1375.169996] Lustre: server umount MGS complete [ 1378.761297] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 14:41:46 (1777574506) [ 1387.744395] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 1390.759913] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 1401.698683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1402.942797] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 1406.716261] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1406.803065] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 1406.819694] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1406.853687] Lustre: lustre-MDT0001: new disk, initializing [ 1406.879125] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1406.889865] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1406.892280] Lustre: 88482:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 1406.894953] LustreError: 88482:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 1408.184562] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 1412.159919] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1412.246756] Lustre: lustre-OST0000: new disk, initializing [ 1412.248300] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1412.269983] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1414.037737] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 1417.110654] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1418.784772] Lustre: 89887:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@0000000010f1e9f5 x1863921917002496/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1777574563 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1419.348415] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1419.352453] Lustre: MGS: Client faa483c1-ddd4-4025-900d-2ad34d94d6c4 (at 0@lo) reconnecting [ 1419.365976] Lustre: 90162:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1419.375359] LustreError: 88468:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 1419.378806] LustreError: 88468:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 1419.390178] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1419.431936] Lustre: lustre-MDT0000: new disk, initializing [ 1419.484505] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1420.512680] Lustre: 90187:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000ac317407 x1863921917007872/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1777574564 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1420.668518] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 1420.832484] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1427.874865] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1427.877433] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1429.602849] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 1429.606811] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 1429.667840] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 1450.465262] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1450.467494] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1450.471286] Lustre: Skipped 1 previous similar message [ 1450.473079] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1455.255100] Lustre: server umount lustre-OST0000 complete [ 1456.609299] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1456.609448] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1456.615173] Lustre: Skipped 2 previous similar messages [ 1461.728967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1461.731645] Lustre: Skipped 5 previous similar messages [ 1462.590536] Lustre: server umount lustre-MDT0000 complete [ 1478.112120] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1478.185394] Lustre: server umount lustre-MDT0001 complete [ 1479.432592] Lustre: server umount MGS complete [ 1482.967406] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 14:43:30 (1777574610) [ 1488.424178] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 1489.430604] Key type lgssc unregistered [ 1489.562317] LNet: 92650:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1489.564706] LNetError: 92650:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1489.577276] LNet: Removed LNI 192.168.202.138@tcp [ 1489.852418] Key type .llcrypt unregistered [ 1489.853817] Key type ._llcrypt unregistered [ 1496.782388] Key type ._llcrypt registered [ 1496.783339] Key type .llcrypt registered [ 1496.817788] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 1497.153650] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1497.180085] alg: No test for adler32 (adler32-zlib) [ 1498.027904] Lustre: Lustre: Build Version: 2.17.52_126_g9629464 [ 1498.105909] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 1498.151183] Key type lgssc registered [ 1500.032231] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1502.513266] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1502.520143] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1503.668063] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1504.865859] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 1507.590710] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1507.710802] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1508.950107] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 1510.584718] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1511.730593] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1513.912455] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1513.954572] LustreError: 95425:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1513.960597] LustreError: 95425:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 1514.030749] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1515.842355] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 1517.551053] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1518.118131] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 1528.292073] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1528.295206] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1528.300181] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1528.303315] Lustre: Skipped 1 previous similar message [ 1532.302534] Lustre: server umount lustre-OST0000 complete [ 1538.531432] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1538.531988] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1538.532500] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1538.535862] Lustre: Skipped 2 previous similar messages [ 1538.538510] LustreError: Skipped 1 previous similar message [ 1539.667964] Lustre: server umount lustre-MDT0000 complete [ 1540.983540] LustreError: 95449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777574669 with bad export cookie 14018626949353074328 [ 1540.984429] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1540.988383] LustreError: 95449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1541.107592] Lustre: server umount lustre-MDT0001 complete [ 1544.509310] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 1545.484637] Key type lgssc unregistered [ 1545.626473] LNet: 97007:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1545.629593] LNetError: 97007:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1545.638312] LNet: Removed LNI 192.168.202.138@tcp [ 1545.941126] Key type .llcrypt unregistered [ 1545.942619] Key type ._llcrypt unregistered [ 1552.394333] Key type ._llcrypt registered [ 1552.395695] Key type .llcrypt registered [ 1552.440359] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 1552.748404] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1552.760150] alg: No test for adler32 (adler32-zlib) [ 1553.612581] Lustre: Lustre: Build Version: 2.17.52_126_g9629464 [ 1553.697271] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 1553.741488] Key type lgssc registered [ 1555.616870] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1557.588835] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1557.588835] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1557.588846] LustreError: 98050:0:(tgt_mount.c:2522:server_fill_super()) cfs_race id 716 sleeping [ 1557.591173] LustreError: 98051:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 waking [ 1557.600723] LustreError: 98050:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 awake: rc=4995 [ 1557.604190] LustreError: 98050:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 1557.607948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1557.609518] LustreError: 98050:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1558.782610] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1561.398900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1561.503303] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1562.822306] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 1564.453745] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing lsmod [ 1565.446865] LustreError: 99219:0:(tgt_mount.c:2522:server_fill_super()) cfs_race id 716 sleeping [ 1565.446879] LustreError: 99220:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 waking [ 1565.451879] LustreError: 99219:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 awake: rc=4998 [ 1565.454532] LustreError: 99219:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 1565.457438] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1565.458963] LustreError: 99219:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1565.578719] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1566.688823] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1566.691560] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1566.695507] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1570.786759] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1570.787802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1570.792556] Lustre: Skipped 1 previous similar message [ 1570.794865] Lustre: Skipped 1 previous similar message [ 1572.628219] Lustre: server umount lustre-MDT0000 complete [ 1573.796382] LustreError: 98066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777574702 with bad export cookie 13979682122539195430 [ 1573.800284] LustreError: 98066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1573.800388] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1574.880238] LustreError: 98516:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000002f2a90e2 x1863922149768192/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1574.925219] Lustre: server umount lustre-MDT0001 complete [ 1586.389727] Lustre: server umount lustre-OST0000 complete [ 1589.050264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1589.208601] LustreError: 100281:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1589.216111] LustreError: 100281:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 1589.228864] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1590.453179] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 1593.299598] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1593.417484] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1594.778794] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 1596.466485] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1597.632183] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1600.996965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1604.958371] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 1608.570656] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1614.510235] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 1618.916396] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1618.928494] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1618.952254] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1618.955859] Lustre: Skipped 1 previous similar message [ 1619.962348] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1620.707694] Lustre: server umount lustre-OST0000 complete [ 1624.033346] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1624.041437] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1624.054101] Lustre: Skipped 1 previous similar message [ 1624.065062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1624.067659] Lustre: Skipped 2 previous similar messages [ 1630.178294] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1630.188179] Lustre: Skipped 3 previous similar messages [ 1638.368339] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1638.623098] Lustre: server umount lustre-MDT0000 complete [ 1640.418142] LustreError: 100282:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1640.426705] LustreError: 100282:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 1641.965050] LustreError: 100262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777574770 with bad export cookie 13979682122539196102 [ 1641.970224] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1641.979887] LustreError: 100262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1642.470318] Lustre: server umount lustre-MDT0001 complete [ 1650.556658] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 1653.474147] Key type lgssc unregistered [ 1653.769629] LNet: 103452:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1653.777798] LNetError: 103452:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1653.805075] LNet: Removed LNI 192.168.202.138@tcp [ 1654.509678] Key type .llcrypt unregistered [ 1654.511276] Key type ._llcrypt unregistered [ 1677.314454] Key type ._llcrypt registered [ 1677.317169] Key type .llcrypt registered [ 1677.424708] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 1678.882418] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1678.986196] alg: No test for adler32 (adler32-zlib) [ 1680.228680] Lustre: Lustre: Build Version: 2.17.52_126_g9629464 [ 1680.575048] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 1680.812433] Key type lgssc registered [ 1684.058108] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1689.715409] hrtimer: interrupt took 26249715 ns [ 1693.331277] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1693.359196] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1694.951787] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1700.535481] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 1705.461583] LustreError: 104845:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1705.491483] LustreError: 104845:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 1709.061622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1709.291651] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1712.797894] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 1717.901791] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1721.674147] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1727.314366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1727.558261] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1731.939662] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 1736.288438] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1739.823478] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:67 to 0x240000401:97) [ 1745.874448] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 1748.143370] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 1754.081090] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1754.088185] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1754.110581] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1755.109740] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1755.110936] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1755.116725] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1756.970746] Lustre: server umount lustre-OST0000 complete [ 1760.226391] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1760.236693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1760.241224] Lustre: Skipped 1 previous similar message [ 1760.260982] Lustre: Skipped 2 previous similar messages [ 1765.349041] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1765.354646] Lustre: Skipped 1 previous similar message [ 1766.263756] Lustre: server umount lustre-MDT0000 complete [ 1769.272546] LustreError: 104827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777574897 with bad export cookie 16555154547800966658 [ 1769.284086] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1769.287366] LustreError: 104827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1769.645102] Lustre: server umount lustre-MDT0001 complete [ 1777.021779] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 1779.450683] Key type lgssc unregistered [ 1779.670557] LNet: 108123:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1779.678259] LNetError: 108123:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1779.706757] LNet: Removed LNI 192.168.202.138@tcp [ 1780.346253] Key type .llcrypt unregistered [ 1780.352973] Key type ._llcrypt unregistered [ 1798.645530] Key type ._llcrypt registered [ 1798.650453] Key type .llcrypt registered [ 1798.934602] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 1799.855941] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1799.890823] alg: No test for adler32 (adler32-zlib) [ 1802.101378] Lustre: Lustre: Build Version: 2.17.52_126_g9629464 [ 1802.450966] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 1802.882227] Key type lgssc registered [ 1806.789451] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1822.333824] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1822.360108] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1824.542742] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1832.117883] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 1833.954152] LustreError: 109330:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1834.020490] LustreError: 109330:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 1839.212579] LustreError: 109331:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1844.199971] LustreError: 109330:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1848.565392] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1849.065467] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1855.597891] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 1861.832600] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1866.678803] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1874.082224] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1874.526652] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1881.200850] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 1887.106395] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1895.040208] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 1907.184524] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 1907.196204] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1907.225308] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1910.753104] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1910.760898] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1913.261873] Lustre: server umount lustre-OST0000 complete [ 1921.015434] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1921.017507] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1921.045978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1921.055415] Lustre: Skipped 1 previous similar message [ 1921.070382] Lustre: Skipped 2 previous similar messages [ 1923.706744] Lustre: server umount lustre-MDT0000 complete [ 1926.114583] LustreError: 109327:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1926.128582] LustreError: 109327:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 1928.141849] LustreError: 109310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777575056 with bad export cookie 3857360020550562439 [ 1928.158384] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1928.162256] LustreError: 109310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1928.837590] Lustre: server umount lustre-MDT0001 complete [ 1941.489272] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 1944.432932] Key type lgssc unregistered [ 1944.759729] LNet: 112532:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1944.767403] LNetError: 112532:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1944.798851] LNet: Removed LNI 192.168.202.138@tcp [ 1945.801953] Key type .llcrypt unregistered [ 1945.805412] Key type ._llcrypt unregistered [ 1973.210364] Key type ._llcrypt registered [ 1973.212351] Key type .llcrypt registered [ 1973.393080] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 1975.273536] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1975.337813] alg: No test for adler32 (adler32-zlib) [ 1976.457617] Lustre: Lustre: Build Version: 2.17.52_126_g9629464 [ 1976.723689] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 1976.941368] Key type lgssc registered [ 1979.884537] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1988.284409] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1988.321305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1989.967584] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1994.359217] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 2000.365554] LustreError: 113933:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2000.387831] LustreError: 113933:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 2004.001199] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2004.504852] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2010.110837] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 2017.055457] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2022.035875] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2030.514255] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2030.879990] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2035.016129] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 2037.858475] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 2043.342360] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2054.222210] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 2057.748797] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 2057.751379] Lustre: Skipped 2 previous similar messages [ 2061.921533] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 2061.932444] Lustre: Skipped 2 previous similar messages [ 2063.010704] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 2063.064926] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2071.876201] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.38@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 2071.892459] Lustre: Skipped 2 previous similar messages [ 2072.182022] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.38@tcp 192.168.0.[2,10]@tcp [ 2077.412154] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.38/24@tcp in log lustre-MDT0000 [ 2077.423737] Lustre: Skipped 2 previous similar messages [ 2081.924647] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.38/24@tcp [ 2081.936444] Lustre: Skipped 1 previous similar message [ 2090.470160] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2090.472048] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2090.496061] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2090.536364] Lustre: Skipped 1 previous similar message [ 2090.554430] Lustre: Skipped 1 previous similar message [ 2095.593201] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2095.609188] Lustre: Skipped 1 previous similar message [ 2096.035895] Lustre: server umount lustre-OST0000 complete [ 2101.729949] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2101.749253] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2101.797139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2105.827904] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2105.831457] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2105.855733] Lustre: Skipped 1 previous similar message [ 2107.760115] Lustre: server umount lustre-MDT0000 complete [ 2110.945496] LustreError: 113928:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2110.960384] LustreError: 113928:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 2114.055146] LustreError: 113913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777575242 with bad export cookie 1330795428375864801 [ 2114.056211] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2114.064177] LustreError: 113913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2115.067044] Lustre: server umount lustre-MDT0001 complete [ 2128.739767] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 2131.959454] Key type lgssc unregistered [ 2132.382194] LNet: 118224:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2132.415319] LNetError: 118224:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2133.482458] LNet: Removed LNI 192.168.202.138@tcp [ 2134.690591] Key type .llcrypt unregistered [ 2134.693521] Key type ._llcrypt unregistered [ 2166.574247] Key type ._llcrypt registered [ 2166.576353] Key type .llcrypt registered [ 2166.711097] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 2168.603557] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2168.646762] alg: No test for adler32 (adler32-zlib) [ 2169.745412] Lustre: Lustre: Build Version: 2.17.52_126_g9629464 [ 2169.981920] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 2170.220923] Key type lgssc registered [ 2173.261252] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2181.661909] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2181.687383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2183.269146] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 2183.308495] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.38/24@tcp [ 2183.316762] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2189.179950] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 2192.879942] LustreError: 119771:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2192.900608] LustreError: 119771:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 2198.003594] LustreError: 119770:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2198.925261] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2199.452568] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2199.489511] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.38/24@tcp [ 2199.570540] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2204.697723] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 2211.017463] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2215.126612] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2222.803503] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2223.199870] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2228.359706] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:169 to 0x240000401:193) [ 2229.706336] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 2235.793414] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2249.195376] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2249.203070] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2249.215821] Lustre: Skipped 1 previous similar message [ 2249.244545] Lustre: Skipped 1 previous similar message [ 2253.054995] Lustre: server umount lustre-OST0000 complete [ 2259.425448] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2259.450205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2264.032605] Lustre: server umount lustre-MDT0000 complete [ 2264.548917] LustreError: 119767:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2264.568373] LustreError: 119767:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 2268.045829] LustreError: 119750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777575396 with bad export cookie 17128511762199666560 [ 2268.051779] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2268.069105] LustreError: 119750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2268.518591] Lustre: server umount lustre-MDT0001 complete [ 2280.555555] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 2283.774960] Key type lgssc unregistered [ 2284.055694] LNet: 123001:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2284.059360] LNetError: 123001:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2284.069840] LNet: Removed LNI 192.168.202.138@tcp [ 2284.814422] Key type .llcrypt unregistered [ 2284.818472] Key type ._llcrypt unregistered [ 2312.564593] Key type ._llcrypt registered [ 2312.571615] Key type .llcrypt registered [ 2312.697573] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 2330.191294] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 2331.778466] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2331.807092] alg: No test for adler32 (adler32-zlib) [ 2332.885247] Lustre: Lustre: Build Version: 2.17.52_126_g9629464 [ 2333.229119] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 2333.453597] Key type lgssc registered [ 2336.324706] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2375.941094] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2375.984857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2377.315289] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2377.356968] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2377.440366] Lustre: lustre-MDT0000: new disk, initializing [ 2377.585779] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2377.620216] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2381.918619] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 2392.043286] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2392.136994] Lustre: 126368:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2392.182821] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2392.185038] Lustre: Skipped 1 previous similar message [ 2392.297539] Lustre: lustre-MDT0001: new disk, initializing [ 2392.418578] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2392.460917] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2392.484755] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2397.344874] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 2405.257674] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2408.871471] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2416.660817] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2417.130791] Lustre: lustre-OST0000: new disk, initializing [ 2417.145894] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2417.312066] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2424.991351] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2424.999450] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2425.173895] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2425.375669] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 2434.218949] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2440.322889] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2440.737290] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2447.977334] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2448.196502] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2456.034752] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2456.035846] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2456.048243] LustreError: Skipped 1 previous similar message [ 2456.050411] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2456.081771] Lustre: Skipped 1 previous similar message [ 2457.957899] Lustre: server umount lustre-OST0000 complete [ 2464.227376] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2464.238069] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2464.248503] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2464.250686] Lustre: Skipped 1 previous similar message [ 2466.274541] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2466.276197] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2466.293188] Lustre: Skipped 1 previous similar message [ 2469.178535] Lustre: server umount lustre-MDT0000 complete [ 2473.063878] LustreError: 126361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777575601 with bad export cookie 10606747078837777039 [ 2473.086404] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2473.583544] Lustre: server umount lustre-MDT0001 complete [ 2483.939895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2484.604176] LustreError: 130180:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2484.635032] LustreError: 130180:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 2484.741653] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2489.826461] LustreError: 130181:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2491.503980] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 2493.931043] LustreError: 130180:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2499.043917] LustreError: 130181:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2501.912653] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2502.459272] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2507.693235] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 2513.432098] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2518.357250] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2526.672468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2527.222305] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2534.498599] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 2540.158654] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2545.015859] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2545.272120] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2550.564685] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2550.821359] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2561.881865] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2562.095552] Lustre: lustre-OST0001: new disk, initializing [ 2562.098312] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2562.216423] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2568.661493] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 2568.869535] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2568.881584] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2569.041702] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2577.457721] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2577.730364] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2583.582967] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2583.785803] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2587.797298] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9fa0d112c000.ost_server_uuid 50 [ 2589.231872] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9fa0d112c000.ost_server_uuid in IDLE state after 0 sec [ 2599.570935] Lustre: server umount lustre-OST0001 complete [ 2599.904888] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2599.907812] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2599.917850] LustreError: Skipped 1 previous similar message [ 2599.927029] LustreError: 131775:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2599.937494] Lustre: Skipped 1 previous similar message [ 2599.967691] LustreError: 131775:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 2605.039668] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2605.050689] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2605.068348] Lustre: Skipped 1 previous similar message [ 2610.168932] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2610.169419] LustreError: 131775:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2610.178308] Lustre: Skipped 2 previous similar messages [ 2610.191603] LustreError: 131775:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 2617.312381] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2617.562367] Lustre: server umount lustre-OST0000 complete [ 2625.510391] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2625.512879] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2625.516214] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2625.516222] Lustre: Skipped 2 previous similar messages [ 2625.530088] Lustre: Skipped 1 previous similar message [ 2627.744335] Lustre: server umount lustre-MDT0000 complete [ 2630.626826] LustreError: 133773:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2630.640995] LustreError: 133773:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 4 previous similar messages [ 2632.111211] LustreError: 130160:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777575760 with bad export cookie 10606747078837777921 [ 2632.121074] LustreError: 130160:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2632.130421] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2632.715821] Lustre: server umount lustre-MDT0001 complete [ 2653.640056] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 2655.846465] Key type lgssc unregistered [ 2656.101561] LNet: 136032:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2656.105394] LNetError: 136032:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2656.122731] LNet: Removed LNI 192.168.202.138@tcp [ 2656.906161] Key type .llcrypt unregistered [ 2656.908560] Key type ._llcrypt unregistered [ 2698.829211] Key type ._llcrypt registered [ 2698.831624] Key type .llcrypt registered [ 2698.931587] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 2715.175348] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 2716.831676] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2716.870136] alg: No test for adler32 (adler32-zlib) [ 2717.975798] Lustre: Lustre: Build Version: 2.17.52_126_g9629464 [ 2718.287369] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 2718.423105] Key type lgssc registered [ 2721.247888] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2758.376632] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2758.402683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2759.710580] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2759.739401] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2759.847459] Lustre: lustre-MDT0000: new disk, initializing [ 2759.991138] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2760.006410] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2764.509939] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 2774.839741] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2774.976237] Lustre: 140157:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2775.022329] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2775.030746] Lustre: Skipped 1 previous similar message [ 2775.086346] Lustre: lustre-MDT0001: new disk, initializing [ 2775.178602] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2775.236320] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2775.250262] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2779.571897] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 2786.321352] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2789.992261] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2796.920614] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2797.197220] Lustre: lustre-OST0000: new disk, initializing [ 2797.200450] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2797.287767] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2803.025068] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 2803.401519] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2803.420698] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2803.512419] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2811.108944] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2816.675946] Lustre: Failing over lustre-OST0000 [ 2816.869411] Lustre: server umount lustre-OST0000 complete [ 2835.424820] Lustre: 138379:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777575947/real 1777575947] req@00000000ee79dd8b x1863923370328448/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1777575963 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2835.431597] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2835.452866] Lustre: 138379:0:(client.c:2479:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 2835.501590] Lustre: Skipped 1 previous similar message [ 2837.378403] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2837.733738] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2837.752490] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2839.781832] Lustre: 138378:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777575952/real 1777575952] req@000000001c2154e2 x1863923370329856/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1777575968 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2839.825574] Lustre: 138378:0:(client.c:2479:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 2844.155563] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 2844.835242] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2845.136227] Lustre: 138379:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777575957/real 1777575957] req@0000000045d3e36f x1863923370330880/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1777575973 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2845.930744] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 2845.935321] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2848.225142] Lustre: Failing over lustre-MDT0000 [ 2848.527967] Lustre: server umount lustre-MDT0000 complete [ 2850.272151] Lustre: 138378:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777575962/real 1777575962] req@00000000428439c4 x1863923370332032/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1777575978 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2850.274482] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2850.307172] Lustre: 138378:0:(client.c:2479:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2850.327538] Lustre: Skipped 1 previous similar message [ 2855.396950] LustreError: 140167:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2855.422320] LustreError: 140167:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 7 previous similar messages [ 2856.113787] LustreError: 143038:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2860.515087] LustreError: 140167:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2860.532872] LustreError: 140167:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 2865.634645] Lustre: 138379:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777575978/real 1777575978] req@000000007049ad2a x1863923370341888/t0(0) o400->MGC192.168.202.138@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1777575994 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2865.645860] LustreError: 140166:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2865.666769] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2865.696679] LustreError: 140166:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 2866.925622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2870.764348] LustreError: 140167:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2870.776538] LustreError: 140167:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 4 previous similar messages [ 2877.212088] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2877.272570] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2881.715702] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2881.805121] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 2882.544740] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2882.554756] Lustre: Skipped 1 previous similar message [ 2882.613373] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2882.671035] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2892.257619] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2892.261829] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2892.286443] Lustre: Skipped 1 previous similar message [ 2892.304560] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2896.314939] Lustre: server umount lustre-OST0000 complete [ 2903.016882] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2903.017145] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2903.024232] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2903.024244] Lustre: Skipped 1 previous similar message [ 2903.050152] Lustre: Skipped 2 previous similar messages [ 2907.316942] Lustre: server umount lustre-MDT0000 complete [ 2908.131888] LustreError: 140163:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2908.148181] LustreError: 140163:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 9 previous similar messages [ 2911.288530] LustreError: 140148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777576039 with bad export cookie 15424135341301041559 [ 2911.297518] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2911.978655] Lustre: server umount lustre-MDT0001 complete [ 2923.651459] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 2926.223767] Key type lgssc unregistered [ 2926.511803] LNet: 145387:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2926.523075] LNetError: 145387:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2927.595035] LNet: Removed LNI 192.168.202.138@tcp [ 2928.420243] Key type .llcrypt unregistered [ 2928.422347] Key type ._llcrypt unregistered [ 2964.726413] Key type ._llcrypt registered [ 2964.728861] Key type .llcrypt registered [ 2964.820669] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 2965.683576] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2965.769081] alg: No test for adler32 (adler32-zlib) [ 2966.850606] Lustre: Lustre: Build Version: 2.17.52_126_g9629464 [ 2967.036364] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 2967.178227] Key type lgssc registered [ 2970.173799] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2979.945588] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2979.970262] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2981.419536] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2986.082771] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 2991.593815] LustreError: 146893:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2991.606359] LustreError: 146893:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 2993.568963] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2996.707992] LustreError: 146892:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2998.458507] LustreError: 146921:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2998.464779] LustreError: 146921:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2998.471601] LustreError: 146921:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 17, retries 0, failed: rc = -5 [ 2998.698990] Lustre: server umount lustre-MDT0000 complete [ 3012.000409] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 3020.108428] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 3052.353357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3052.586455] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3052.624622] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3052.732351] Lustre: lustre-MDT0000: new disk, initializing [ 3052.802087] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3052.816727] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3056.889057] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 3067.080255] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3067.167746] Lustre: 150078:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3067.204762] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3067.206931] Lustre: Skipped 1 previous similar message [ 3067.246827] Lustre: lustre-MDT0001: new disk, initializing [ 3067.325744] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3067.360324] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3067.374244] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3071.210620] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 3077.840551] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3082.196642] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3089.067974] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3089.289377] Lustre: lustre-OST0000: new disk, initializing [ 3089.292429] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3089.409793] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3095.368236] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 3098.668830] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3098.694032] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3098.810977] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3102.415941] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3105.903979] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 3107.618781] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 15:10:34 (1777576234) [ 3323.360773] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3323.367537] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3323.378300] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3324.390979] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3324.398087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3324.404084] Lustre: Skipped 1 previous similar message [ 3324.422269] Lustre: Skipped 2 previous similar messages [ 3326.923996] Lustre: server umount lustre-MDT0000 complete [ 3329.507239] LustreError: 150090:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3329.518949] LustreError: 150090:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 3330.127947] LustreError: 150071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777576458 with bad export cookie 3303144473537609947 [ 3330.136942] LustreError: 150071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3330.143934] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3334.624707] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3334.625055] LustreError: 150087:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3334.631813] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3336.383464] Lustre: server umount lustre-MDT0001 complete [ 3345.826717] Lustre: server umount lustre-OST0000 complete [ 3353.897799] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 3363.954927] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 3398.836034] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3399.090569] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3399.155385] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3399.257609] Lustre: lustre-MDT0000: new disk, initializing [ 3399.355106] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3399.379200] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3403.965901] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 3414.786403] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3414.948872] Lustre: 156323:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3415.034716] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3415.043775] Lustre: Skipped 1 previous similar message [ 3415.164371] Lustre: lustre-MDT0001: new disk, initializing [ 3415.296385] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3415.353120] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3415.364042] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3419.225761] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 3426.977666] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3430.384368] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3437.347957] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3437.522919] Lustre: lustre-OST0000: new disk, initializing [ 3437.526273] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3437.609187] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3443.921977] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 3444.292659] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3444.303422] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3444.407406] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3450.723039] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3455.503425] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3455.712859] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3460.606977] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3460.803679] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3465.190832] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3465.193252] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3465.196321] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3465.196329] Lustre: Skipped 1 previous similar message [ 3465.201703] LustreError: Skipped 1 previous similar message [ 3465.239939] Lustre: Skipped 1 previous similar message [ 3468.682142] Lustre: server umount lustre-OST0000 complete [ 3475.425958] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3475.444191] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3475.450240] Lustre: Skipped 2 previous similar messages [ 3478.695260] Lustre: server umount lustre-MDT0000 complete [ 3480.545503] LustreError: 158332:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3480.557433] LustreError: 158332:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 3482.368812] LustreError: 156316:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777576610 with bad export cookie 3303144473537768777 [ 3482.373747] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3482.378474] LustreError: 156316:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3482.673547] Lustre: server umount lustre-MDT0001 complete [ 3491.053764] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 15:16:57 (1777576617) [ 3509.665714] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 3519.527418] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 3555.490376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3555.765781] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3555.828383] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3555.960423] Lustre: lustre-MDT0000: new disk, initializing [ 3556.088819] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3556.143543] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3560.763610] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 3572.432547] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3572.578037] Lustre: 162641:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3572.846833] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3576.946831] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 3583.769441] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3587.856629] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3594.516341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3594.745409] Lustre: lustre-OST0000: new disk, initializing [ 3594.748203] Lustre: Skipped 1 previous similar message [ 3594.751890] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3594.759796] Lustre: Skipped 2 previous similar messages [ 3601.323504] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 3603.513627] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3603.521278] Lustre: Skipped 1 previous similar message [ 3603.525555] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3603.640346] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3608.807682] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3617.468673] Lustre: server umount lustre-OST0000 complete [ 3623.912602] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3623.918142] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3623.929020] Lustre: Skipped 1 previous similar message [ 3623.948595] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3631.784065] Lustre: server umount lustre-MDT0000 complete [ 3634.656104] Lustre: 146202:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777576747/real 1777576747] req@0000000018bb542f x1863923632142592/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1777576763 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3634.660299] LustreError: 162647:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3634.713608] LustreError: 162647:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 3635.678671] LustreError: 162632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777576764 with bad export cookie 3303144473537769659 [ 3635.696227] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3636.020657] Lustre: server umount lustre-MDT0001 complete [ 3644.823883] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 15:19:31 (1777576771) [ 3661.507205] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 3671.266959] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 3712.488420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3712.884735] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3712.931738] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3713.134693] Lustre: lustre-MDT0000: new disk, initializing [ 3713.279649] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3713.282506] Lustre: Skipped 2 previous similar messages [ 3713.324915] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3719.791586] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 3732.381868] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3732.497743] Lustre: 168565:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3732.783922] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3737.763982] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 3745.422403] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3749.505620] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3758.864284] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3765.926922] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3766.169357] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3767.481947] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 3776.331257] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3786.723656] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3786.727621] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3786.739813] Lustre: Skipped 2 previous similar messages [ 3786.747330] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3786.763588] Lustre: Skipped 5 previous similar messages [ 3789.534483] Lustre: server umount lustre-OST0000 complete [ 3794.405139] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3799.624341] Lustre: server umount lustre-MDT0000 complete [ 3802.086128] LustreError: 168574:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3802.107119] LustreError: 168574:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 3803.513981] LustreError: 168558:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777576931 with bad export cookie 3303144473537770744 [ 3803.518161] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3803.522257] LustreError: 168558:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 3803.880593] Lustre: server umount lustre-MDT0001 complete [ 3813.020643] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 3816.188024] Key type lgssc unregistered [ 3816.546299] LNet: 172042:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3816.553684] LNetError: 172042:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3817.578272] LNet: Removed LNI 192.168.202.138@tcp [ 3818.652851] Key type .llcrypt unregistered [ 3818.657776] Key type ._llcrypt unregistered [ 3848.193791] Key type ._llcrypt registered [ 3848.196861] Key type .llcrypt registered [ 3848.417374] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 3850.268942] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3850.294838] alg: No test for adler32 (adler32-zlib) [ 3851.368876] Lustre: Lustre: Build Version: 2.17.52_126_g9629464 [ 3851.576892] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 3851.729405] Key type lgssc registered [ 3854.446614] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3862.802736] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3862.839322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3864.361946] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3868.744992] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 3873.769710] LustreError: 173441:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3873.787027] LustreError: 173441:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 3878.640867] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3878.883117] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3879.067755] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3882.858947] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 3889.316067] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3892.838981] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3899.544587] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3899.918164] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3906.489231] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 3910.135372] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 3913.640845] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3935.716676] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3935.716936] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3935.730343] LustreError: Skipped 1 previous similar message [ 3935.732280] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3935.740042] Lustre: Skipped 1 previous similar message [ 3935.753555] Lustre: Skipped 1 previous similar message [ 3939.633720] Lustre: server umount lustre-OST0000 complete [ 3945.441887] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3945.453119] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3945.478459] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3949.855298] Lustre: server umount lustre-MDT0000 complete [ 3951.074864] LustreError: 173441:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3951.101562] LustreError: 173441:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 3953.988110] LustreError: 173422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777577082 with bad export cookie 17873838593772592917 [ 3954.004771] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3954.007453] LustreError: 173422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3954.468855] Lustre: server umount lustre-MDT0001 complete [ 3966.077595] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 3969.413362] Key type lgssc unregistered [ 3969.735281] LNet: 176640:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3969.749359] LNetError: 176640:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3969.779243] LNet: Removed LNI 192.168.202.138@tcp [ 3970.680522] Key type .llcrypt unregistered [ 3970.682584] Key type ._llcrypt unregistered [ 3996.314925] Key type ._llcrypt registered [ 3996.321690] Key type .llcrypt registered [ 3996.477840] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 3997.927871] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3997.962321] alg: No test for adler32 (adler32-zlib) [ 3999.071967] Lustre: Lustre: Build Version: 2.17.52_126_g9629464 [ 3999.285081] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 3999.405887] Key type lgssc registered [ 4002.218926] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4009.248408] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4009.258813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4010.910981] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4015.217250] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 4021.232535] LustreError: 178043:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4021.262094] LustreError: 178043:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 4022.326808] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4022.597746] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4026.857474] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 4032.169639] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4037.573749] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4045.201751] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4045.504109] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4051.575876] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 4055.750080] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 4057.186063] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4063.939098] Lustre: lustre-OST0000: Not available for connect from 192.168.202.38@tcp (stopping) [ 4066.273831] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4066.278503] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4066.292869] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4066.307889] Lustre: Skipped 1 previous similar message [ 4069.042886] Lustre: lustre-OST0000: Not available for connect from 192.168.202.38@tcp (stopping) [ 4069.048976] Lustre: Skipped 1 previous similar message [ 4069.746701] Lustre: server umount lustre-OST0000 complete [ 4075.120613] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9fa0c2bde800.ost_server_uuid 50 [ 4076.355825] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 4090.850787] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4090.867164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4090.880184] Lustre: Skipped 1 previous similar message [ 4095.969646] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4095.983935] Lustre: Skipped 1 previous similar message [ 4096.894380] Lustre: server umount lustre-MDT0000 complete [ 4100.697578] LustreError: 178021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777577229 with bad export cookie 2670230402306525297 [ 4100.707907] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4100.722426] LustreError: 178021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4101.120757] Lustre: server umount lustre-MDT0001 complete [ 4110.038201] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 15:27:16 (1777577236) [ 4117.110938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4117.486812] LustreError: 181520:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4117.579677] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4121.594246] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 4122.597250] LustreError: 181521:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4126.701544] LustreError: 181520:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4130.454528] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4131.041995] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4135.713176] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 4141.669934] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4146.423946] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4154.070415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4154.468343] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4161.314129] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 4161.667719] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4166.590853] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4174.325157] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4174.643257] Lustre: lustre-OST0001: new disk, initializing [ 4174.650775] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4174.742374] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4182.087945] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4182.105246] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4182.263665] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4182.274481] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 4190.273435] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4196.074806] Lustre: lustre-OST0000: Not available for connect from 192.168.202.38@tcp (stopping) [ 4197.871683] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4197.906881] Lustre: Skipped 2 previous similar messages [ 4197.915079] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4207.072165] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4207.288097] Lustre: server umount lustre-OST0000 complete [ 4208.107430] LustreError: 183119:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4208.120069] LustreError: 183119:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 4213.482644] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4213.656603] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 4216.506092] LustreError: 183116:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4216.526469] LustreError: 183116:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 4218.384815] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4218.560205] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 4233.700358] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4233.702743] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4233.705534] LustreError: 183116:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4233.705545] LustreError: 183116:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 8 previous similar messages [ 4233.716091] LustreError: Skipped 1 previous similar message [ 4233.717360] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4233.742426] Lustre: Skipped 1 previous similar message [ 4233.797250] Lustre: Skipped 7 previous similar messages [ 4238.682700] Lustre: server umount lustre-OST0001 complete [ 4243.429498] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4243.439475] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4251.864215] Lustre: server umount lustre-MDT0000 complete [ 4257.207159] LustreError: 181500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777577385 with bad export cookie 2670230402306526368 [ 4257.212295] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4257.226113] LustreError: 181500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4257.650153] Lustre: server umount lustre-MDT0001 complete [ 4282.901880] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 15:30:09 (1777577409) [ 4289.495297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4289.888971] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4289.972450] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4290.230134] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4294.812472] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 4305.337190] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4305.484131] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4305.531440] Lustre: 187313:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4311.148130] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 4318.354785] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4322.802364] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4329.491363] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4329.651547] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4335.904540] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 4338.345972] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 4341.662041] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4347.970417] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4348.023629] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4355.031766] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 4362.332453] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4368.068308] Lustre: lustre-OST0000: Not available for connect from 192.168.202.38@tcp (stopping) [ 4368.074947] Lustre: Skipped 4 previous similar messages [ 4369.377254] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4369.389463] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4369.407601] Lustre: Skipped 2 previous similar messages [ 4371.314393] Lustre: server umount lustre-OST0000 complete [ 4373.172304] LustreError: 188941:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4373.184630] LustreError: 188941:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 4389.870530] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4389.871959] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4389.887602] LustreError: Skipped 2 previous similar messages [ 4389.904600] Lustre: Skipped 1 previous similar message [ 4392.925425] Lustre: server umount lustre-OST0001 complete [ 4403.511886] Lustre: server umount lustre-MDT0000 complete [ 4406.819358] LustreError: 187307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777577535 with bad export cookie 2670230402306527775 [ 4406.825427] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4406.828406] LustreError: 187307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4407.124751] Lustre: server umount lustre-MDT0001 complete [ 4431.512327] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 15:32:37 (1777577557) [ 4449.257271] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 4461.230915] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 4498.568811] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4498.817931] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4498.845476] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4498.957663] Lustre: lustre-MDT0000: new disk, initializing [ 4499.059512] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4499.064223] Lustre: Skipped 3 previous similar messages [ 4499.084614] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4503.957420] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 4515.559291] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4515.663862] Lustre: 194925:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4515.707234] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4515.710410] Lustre: Skipped 1 previous similar message [ 4515.774961] Lustre: lustre-MDT0001: new disk, initializing [ 4515.914895] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4515.935354] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4520.742894] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 4527.930968] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4531.223411] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4538.553940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4538.883933] Lustre: lustre-OST0000: new disk, initializing [ 4538.889666] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4545.752190] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 4547.129109] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4547.141317] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4547.247878] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4553.925084] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4560.003989] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4560.298328] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4567.245846] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4567.527278] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4573.155976] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4573.159407] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4573.171825] Lustre: Skipped 2 previous similar messages [ 4573.177031] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4573.187217] LustreError: Skipped 1 previous similar message [ 4573.215911] Lustre: Skipped 8 previous similar messages [ 4576.193757] Lustre: server umount lustre-OST0000 complete [ 4586.721520] Lustre: server umount lustre-MDT0000 complete [ 4588.517429] LustreError: 194931:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4588.545488] LustreError: 194931:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 13 previous similar messages [ 4591.133374] LustreError: 195713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777577719 with bad export cookie 2670230402306529336 [ 4591.146344] LustreError: 195713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4591.149208] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4591.429143] Lustre: server umount lustre-MDT0001 complete [ 4600.449724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4605.959329] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 4615.462976] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4620.147733] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 4625.440947] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4628.952435] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4635.149425] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4635.511981] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4635.521324] Lustre: Skipped 4 previous similar messages [ 4641.650493] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 4648.845637] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4654.161949] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4654.447276] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4659.913520] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4660.305414] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4664.296061] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4664.335675] Lustre: Skipped 4 previous similar messages [ 4669.343314] Lustre: server umount lustre-OST0000 complete [ 4677.348609] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4683.916399] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4713.166584] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4720.541758] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 4726.438441] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4728.820103] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4728.829763] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4728.830494] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4728.845220] Lustre: Skipped 1 previous similar message [ 4732.010262] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 3 sec [ 4739.046345] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4739.058204] LustreError: Skipped 1 previous similar message [ 4739.070216] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4739.084562] Lustre: Skipped 6 previous similar messages [ 4742.362239] Lustre: server umount lustre-OST0000 complete [ 4753.265694] Lustre: server umount lustre-MDT0000 complete [ 4754.410762] LustreError: 198732:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4754.446646] LustreError: 198732:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 4 previous similar messages [ 4757.377167] LustreError: 198717:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777577885 with bad export cookie 2670230402306530218 [ 4757.392936] LustreError: 198717:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4757.393416] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4757.701738] Lustre: server umount lustre-MDT0001 complete [ 4765.709821] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 15:38:12 (1777577892) [ 4772.485244] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4778.124310] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 4786.871065] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4791.679195] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 4798.124684] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4802.561816] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4810.213507] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4817.434775] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 4823.361951] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4828.482407] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4828.707423] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4833.075603] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4833.284954] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4841.382614] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4841.681875] Lustre: lustre-OST0001: new disk, initializing [ 4841.691972] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4850.091867] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 4851.439764] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4851.474383] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4851.719160] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4860.282547] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4866.500829] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4866.837482] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4872.268977] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4872.639751] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4877.298811] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4877.301303] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4877.315262] LustreError: Skipped 3 previous similar messages [ 4877.350916] Lustre: Skipped 5 previous similar messages [ 4881.345595] Lustre: server umount lustre-OST0001 complete [ 4888.621632] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4888.826762] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 4892.821632] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4892.995487] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 4923.003194] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4923.374239] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4923.384258] Lustre: Skipped 5 previous similar messages [ 4928.497506] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4928.505673] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4928.512479] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4930.578822] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 4937.236238] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4938.934979] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4947.334702] Lustre: server umount lustre-OST0001 complete [ 4971.378881] Lustre: server umount lustre-OST0000 complete [ 4982.565572] Lustre: server umount lustre-MDT0000 complete [ 4986.926450] LustreError: 204165:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777578115 with bad export cookie 2670230402306531184 [ 4986.927103] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4986.943040] LustreError: 204165:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4993.360912] Lustre: server umount lustre-MDT0001 complete [ 5018.883099] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 15:42:25 (1777578145) [ 5027.651108] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5027.899644] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5027.933938] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5032.013500] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 5041.333252] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5041.432487] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5041.458374] Lustre: 212142:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5046.155655] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 5052.162756] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5056.587372] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5064.357128] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5064.633192] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5071.928986] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 5078.335189] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5088.428370] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5088.477830] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5094.974857] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 5101.147458] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5103.363069] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5108.155582] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5108.386398] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5112.945434] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5113.309437] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5117.974244] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9fa0c84f8800.ost_server_uuid 50 [ 5119.973803] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9fa0c84f8800.ost_server_uuid in IDLE state after 0 sec [ 5120.846415] Lustre: Permanently deactivating lustre-OST0001 [ 5120.879479] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 5125.274746] Lustre: Permanently reactivating lustre-OST0001 [ 5125.302173] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 5125.309308] Lustre: Skipped 2 previous similar messages [ 5130.739802] LustreError: lustre-OST0001-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 5130.754248] LustreError: Skipped 5 previous similar messages [ 5130.764939] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5130.770807] Lustre: Skipped 13 previous similar messages [ 5134.313514] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5134.327674] Lustre: Skipped 7 previous similar messages [ 5136.793462] Lustre: server umount lustre-OST0001 complete [ 5139.937226] LustreError: 213761:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5139.964131] LustreError: 213761:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 39 previous similar messages [ 5156.320504] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5156.517591] Lustre: server umount lustre-OST0000 complete [ 5175.264138] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5175.502391] Lustre: server umount lustre-MDT0000 complete [ 5181.027190] LustreError: 212878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777578309 with bad export cookie 2670230402306532472 [ 5181.037796] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5181.046329] LustreError: 212878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5181.578809] Lustre: server umount lustre-MDT0001 complete [ 5212.493086] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 15:45:38 (1777578338) [ 5221.962076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5222.430607] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5222.482813] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5222.501617] Lustre: Skipped 2 previous similar messages [ 5228.584498] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 5240.154744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5240.248910] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5240.313955] Lustre: 218255:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5245.213885] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 5251.190713] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5255.035378] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5262.602233] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5262.927655] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5262.955518] Lustre: Permanently deactivating lustre-OST0000 [ 5262.966475] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 5269.032388] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5270.093716] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 5276.151504] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5284.549673] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5284.685565] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5291.364710] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 5293.403713] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 5301.405281] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5303.592709] Lustre: Permanently reactivating lustre-OST0000 [ 5303.618346] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 5303.626331] Lustre: Skipped 2 previous similar messages [ 5310.603581] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 5310.607567] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 41 seconds. I think it's dead, and I am evicting it. exp 0000000084048bf7, cur 1777578438 deadline 1777578427 last 1777578397 [ 5310.620193] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5310.651369] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5310.674476] Lustre: Skipped 1 previous similar message [ 5310.748536] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5325.365965] Lustre: server umount lustre-OST0001 complete [ 5338.097937] Lustre: server umount lustre-OST0000 complete [ 5350.709470] Lustre: server umount lustre-MDT0000 complete [ 5357.356980] LustreError: 220798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777578485 with bad export cookie 2670230402306534138 [ 5357.363487] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5357.381186] LustreError: 220798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5357.821093] Lustre: server umount lustre-MDT0001 complete [ 5372.966177] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 5377.217499] Key type lgssc unregistered [ 5377.580640] LNet: 222672:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5377.589116] LNetError: 222672:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5377.626895] LNet: Removed LNI 192.168.202.138@tcp [ 5379.127831] Key type .llcrypt unregistered [ 5379.136350] Key type ._llcrypt unregistered [ 5414.710244] Key type ._llcrypt registered [ 5414.712790] Key type .llcrypt registered [ 5414.909251] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 5417.616178] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5417.680233] alg: No test for adler32 (adler32-zlib) [ 5418.753891] Lustre: Lustre: Build Version: 2.17.52_126_g9629464 [ 5418.909141] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 5419.030626] Key type lgssc registered [ 5422.109987] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5434.984858] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5435.008772] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5436.662534] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5441.390546] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 5446.120656] LustreError: 224131:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5446.138591] LustreError: 224131:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 5451.234724] LustreError: 224132:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5452.334704] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5452.635817] Lustre: Found index 1 for lustre-MDT0001, updating log [ 5452.685668] Lustre: 224121:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5452.698574] Lustre: Permanently deactivating lustre-MDT0001 [ 5452.716926] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 5452.865656] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 5452.875247] LustreError: 224163:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 5452.881381] LustreError: 224163:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 5452.887062] LustreError: 224163:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 16, retries 1, failed: rc = -108 [ 5453.263638] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5458.376757] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 5465.009715] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5469.089813] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5477.327254] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5477.762127] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5478.774111] LustreError: 225744:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5480.817064] LustreError: 225747:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5480.835585] LustreError: 225747:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 5480.842227] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 5484.857063] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 5486.056955] LustreError: 225744:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5486.077624] LustreError: 225744:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 5491.375521] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5496.291923] LustreError: 225746:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5496.309101] LustreError: 225746:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 5499.319886] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5499.560520] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5505.020211] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 5505.032225] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 5505.798469] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 5513.630354] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5516.658392] Lustre: Permanently reactivating lustre-MDT0001 [ 5516.683265] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 5516.692600] Lustre: Skipped 2 previous similar messages [ 5520.242111] Lustre: lustre-MDT0001: Client 3333660e-4ae3-4283-8a5e-979640e0afcc (at 192.168.202.38@tcp) reconnecting [ 5525.724725] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5527.863672] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5534.399867] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 5534.631794] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 5539.572758] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 5539.807714] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 5542.762799] Lustre: Permanently deactivating lustre-MDT0001 [ 5542.795947] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 5542.807640] Lustre: Skipped 2 previous similar messages [ 5549.735281] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 5560.812474] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5560.815033] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5560.823673] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5562.343713] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5562.345656] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5562.364024] Lustre: Skipped 1 previous similar message [ 5562.380969] Lustre: Skipped 1 previous similar message [ 5565.645547] Lustre: server umount lustre-MDT0000 complete [ 5567.466484] LustreError: 224132:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5567.484958] LustreError: 224132:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 5570.460597] LustreError: 227226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777578698 with bad export cookie 16906493600907017934 [ 5570.467127] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5570.473419] LustreError: 227226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5570.985517] Lustre: server umount lustre-MDT0001 complete [ 5578.075297] Lustre: server umount lustre-OST0000 complete [ 5591.105248] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 15:51:57 (1777578717) [ 5602.530374] Lustre: server umount lustre-OST0001 complete [ 5610.843471] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 5621.744330] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 5668.273481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5668.606958] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5668.614933] Lustre: Skipped 2 previous similar messages [ 5668.651187] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5668.764696] Lustre: lustre-MDT0000: new disk, initializing [ 5668.899492] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5668.918805] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5674.581458] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 5686.923028] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5687.121842] Lustre: 231774:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5687.264609] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5687.276973] Lustre: Skipped 1 previous similar message [ 5687.470341] Lustre: lustre-MDT0001: new disk, initializing [ 5687.615347] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5687.669600] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5687.687221] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5692.737242] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 5701.115389] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5706.236362] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5714.335218] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5714.683222] Lustre: lustre-OST0000: new disk, initializing [ 5714.709305] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5714.829191] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5722.275167] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 5724.781884] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5724.795789] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5724.946356] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5730.326073] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5732.516565] LustreError: 234105:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 5742.292394] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5742.498595] Lustre: lustre-OST0001: new disk, initializing [ 5742.507207] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5742.560449] LustreError: 234105:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 5742.597847] LustreError: 231780:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 5749.420555] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 5752.688516] LustreError: 231780:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 5752.930643] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5752.939269] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5753.091536] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5757.792995] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5760.087726] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5763.562470] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5763.570673] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5763.592377] Lustre: Skipped 1 previous similar message [ 5763.596607] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5763.596616] Lustre: Skipped 2 previous similar messages [ 5768.708719] LustreError: 233501:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5768.751890] LustreError: 233501:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 5769.089590] Lustre: server umount lustre-OST0001 complete [ 5789.159718] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5789.170602] Lustre: Skipped 1 previous similar message [ 5789.193229] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5789.197593] Lustre: Skipped 2 previous similar messages [ 5794.276293] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5794.287513] Lustre: Skipped 1 previous similar message [ 5795.486466] Lustre: server umount lustre-OST0000 complete [ 5799.394522] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5799.413530] Lustre: Skipped 1 previous similar message [ 5804.516735] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5804.529881] Lustre: Skipped 3 previous similar messages [ 5805.590338] Lustre: server umount lustre-MDT0000 complete [ 5810.077055] LustreError: 231765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777578938 with bad export cookie 16906493600907020538 [ 5810.082913] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5810.090842] LustreError: 231765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5816.792790] Lustre: server umount lustre-MDT0001 complete [ 5829.045470] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 5833.385424] Key type lgssc unregistered [ 5833.900766] LNet: 236421:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5833.910507] LNetError: 236421:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5833.950069] LNet: Removed LNI 192.168.202.138@tcp [ 5835.396884] Key type .llcrypt unregistered [ 5835.398595] Key type ._llcrypt unregistered [ 5886.474524] Key type ._llcrypt registered [ 5886.476329] Key type .llcrypt registered [ 5886.598078] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 5887.947553] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5887.972187] alg: No test for adler32 (adler32-zlib) [ 5889.124928] Lustre: Lustre: Build Version: 2.17.52_126_g9629464 [ 5889.499934] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 5889.782370] Key type lgssc registered [ 5892.898623] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5902.789294] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5902.815586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5904.157941] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5904.177951] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5904.302926] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5909.566657] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 5920.745058] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5920.992640] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5921.103322] Lustre: 238616:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5921.618519] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5926.998523] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 5934.593458] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5939.775173] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5947.891197] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5948.041967] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5948.241095] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5955.195638] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 5956.882911] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 5962.236783] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6018.533747] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6018.535929] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6018.539955] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6018.551287] Lustre: Skipped 1 previous similar message [ 6022.253449] Lustre: server umount lustre-OST0000 complete [ 6027.459694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 6041.801646] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6042.233819] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6049.247917] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6056.054612] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6059.631206] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6059.657417] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6059.657730] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6059.683312] Lustre: Skipped 1 previous similar message [ 6095.848351] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6095.862927] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6095.867577] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6095.867585] Lustre: Skipped 1 previous similar message [ 6095.900334] Lustre: Skipped 1 previous similar message [ 6100.963134] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6100.972496] Lustre: Skipped 2 previous similar messages [ 6101.511537] Lustre: server umount lustre-OST0000 complete [ 6111.201370] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6111.223742] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6111.235553] Lustre: Skipped 1 previous similar message [ 6112.752196] Lustre: server umount lustre-MDT0000 complete [ 6117.175694] LustreError: 240253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777579245 with bad export cookie 202059363450016668 [ 6117.176749] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6117.190506] LustreError: 240253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6117.503334] Lustre: server umount lustre-MDT0001 complete [ 6128.947450] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 6132.287717] Key type lgssc unregistered [ 6132.659596] LNet: 243570:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6132.676428] LNetError: 243570:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6132.711227] LNet: Removed LNI 192.168.202.138@tcp [ 6133.861584] Key type .llcrypt unregistered [ 6133.869194] Key type ._llcrypt unregistered [ 6164.980446] Key type ._llcrypt registered [ 6164.991138] Key type .llcrypt registered [ 6165.158190] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 6167.182168] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6167.192674] alg: No test for adler32 (adler32-zlib) [ 6169.321565] Lustre: Lustre: Build Version: 2.17.52_126_g9629464 [ 6169.888095] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 6170.213357] Key type lgssc registered [ 6175.207297] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6188.546554] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6188.565038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6190.823384] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6196.413137] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6201.324839] LustreError: 244985:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6201.349473] LustreError: 244985:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 6206.069776] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6206.445481] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6210.917229] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6217.357600] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6221.931947] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6232.222307] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6232.890234] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6236.000877] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 6241.487353] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6249.129949] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6282.226857] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6282.230917] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6282.278081] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6282.283432] Lustre: Skipped 1 previous similar message [ 6284.624620] Lustre: server umount lustre-OST0000 complete [ 6292.454879] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6292.468384] Lustre: Skipped 1 previous similar message [ 6292.482897] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6292.490993] Lustre: Skipped 1 previous similar message [ 6295.885778] Lustre: server umount lustre-MDT0000 complete [ 6297.596965] LustreError: 245958:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6297.633827] LustreError: 245958:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 6301.431545] LustreError: 247913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777579429 with bad export cookie 3175200403851112715 [ 6301.441488] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6301.458228] LustreError: 247913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6302.385802] Lustre: server umount lustre-MDT0001 complete [ 6321.989330] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 6326.370281] Key type lgssc unregistered [ 6326.843565] LNet: 248956:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6326.868217] LNetError: 248956:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6326.911649] LNet: Removed LNI 192.168.202.138@tcp [ 6328.146368] Key type .llcrypt unregistered [ 6328.147963] Key type ._llcrypt unregistered [ 6350.533770] Key type ._llcrypt registered [ 6350.539374] Key type .llcrypt registered [ 6350.668381] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 6352.048264] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6352.298157] alg: No test for adler32 (adler32-zlib) [ 6353.323906] Lustre: Lustre: Build Version: 2.17.52_126_g9629464 [ 6353.504204] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 6353.605676] Key type lgssc registered [ 6356.412360] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6364.718286] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6364.760390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6366.456977] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6371.173960] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6375.910744] LustreError: 250160:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6375.943085] LustreError: 250160:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 6380.895890] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6381.033491] LustreError: 250159:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6381.566613] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6386.291421] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6392.538883] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6396.859407] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6404.841800] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6405.433570] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6411.579436] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6412.430040] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6419.116782] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6437.383896] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6437.411469] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6437.412709] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6439.258759] Lustre: server umount lustre-OST0000 complete [ 6447.586556] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6447.597464] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6447.608827] Lustre: Skipped 2 previous similar messages [ 6447.628573] Lustre: Skipped 2 previous similar messages [ 6450.394567] Lustre: server umount lustre-MDT0000 complete [ 6452.710407] LustreError: 250155:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6452.738336] LustreError: 250155:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 6455.213933] LustreError: 250139:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777579583 with bad export cookie 2554336965880988844 [ 6455.230780] LustreError: 250139:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6455.231480] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6455.757327] Lustre: server umount lustre-MDT0001 complete [ 6467.957828] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 6471.308589] Key type lgssc unregistered [ 6471.631897] LNet: 253483:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6471.639106] LNetError: 253483:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6471.667293] LNet: Removed LNI 192.168.202.138@tcp [ 6472.759304] Key type .llcrypt unregistered [ 6472.762384] Key type ._llcrypt unregistered [ 6499.801712] Key type ._llcrypt registered [ 6499.802907] Key type .llcrypt registered [ 6499.918949] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 6500.956947] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6500.988978] alg: No test for adler32 (adler32-zlib) [ 6502.423722] Lustre: Lustre: Build Version: 2.17.52_126_g9629464 [ 6502.852882] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 6503.049411] Key type lgssc registered [ 6506.139131] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6515.893602] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6515.931216] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6517.608611] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6523.168706] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6526.951209] LustreError: 254886:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6526.981231] LustreError: 254886:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 6532.066411] LustreError: 254885:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6535.050374] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6536.025231] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6541.982854] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6548.250742] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6552.642794] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6562.317737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6562.809259] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6569.909334] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6576.939474] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6587.411804] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6622.178156] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6622.223030] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6622.286423] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6623.211428] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6623.227647] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6623.239705] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6626.284247] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6628.222659] Lustre: server umount lustre-OST0000 complete [ 6633.461689] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6633.472780] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6633.504325] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6633.506969] Lustre: Skipped 1 previous similar message [ 6636.539194] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6641.637723] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6641.649323] Lustre: Skipped 3 previous similar messages [ 6647.264175] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6647.823261] Lustre: server umount lustre-MDT0000 complete [ 6650.853644] LustreError: 255781:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6650.895686] LustreError: 255781:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 6652.374275] LustreError: 254866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777579780 with bad export cookie 15526515097648949615 [ 6652.380361] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6652.382438] LustreError: 254866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6652.809912] Lustre: server umount lustre-MDT0001 complete [ 6667.942423] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 6672.521717] Key type lgssc unregistered [ 6672.924240] LNet: 258859:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6672.944429] LNetError: 258859:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6672.962660] LNet: Removed LNI 192.168.202.138@tcp [ 6674.165100] Key type .llcrypt unregistered [ 6674.171821] Key type ._llcrypt unregistered [ 6694.686978] Key type ._llcrypt registered [ 6694.691376] Key type .llcrypt registered [ 6694.802301] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 6695.963286] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6696.179963] alg: No test for adler32 (adler32-zlib) [ 6697.324218] Lustre: Lustre: Build Version: 2.17.52_126_g9629464 [ 6697.665234] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 6697.942887] Key type lgssc registered [ 6700.874512] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6709.817689] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6709.843903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6711.367606] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6716.190700] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6721.505511] LustreError: 260081:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6721.531765] LustreError: 260081:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 6725.904972] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6726.431400] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6731.568900] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6738.706587] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6744.333948] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6754.215891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6754.815035] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6755.852771] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 6762.391293] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6769.370853] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6789.976413] Lustre: server umount lustre-OST0000 complete [ 6796.772489] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6796.776862] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6796.803893] Lustre: Skipped 1 previous similar message [ 6796.827274] Lustre: Skipped 1 previous similar message [ 6801.895135] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6801.904516] Lustre: Skipped 1 previous similar message [ 6805.251046] Lustre: server umount lustre-MDT0000 complete [ 6807.009959] LustreError: 260086:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6807.038381] LustreError: 260086:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 6808.032243] Lustre: 259376:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777579920/real 1777579920] req@0000000065b96318 x1863927543658880/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1777579936 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6808.035995] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6808.076205] Lustre: 259376:0:(client.c:2479:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 6808.130793] LustreError: 260090:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6808.149940] LustreError: 260090:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 6809.745321] LustreError: 260047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777579938 with bad export cookie 59208368277687184 [ 6809.758049] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6809.771367] LustreError: 260047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6810.329285] Lustre: server umount lustre-MDT0001 complete [ 6822.918496] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 6826.144525] Key type lgssc unregistered [ 6826.464555] LNet: 263428:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6826.472983] LNetError: 263428:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6826.487234] LNet: Removed LNI 192.168.202.138@tcp [ 6827.463874] Key type .llcrypt unregistered [ 6827.466352] Key type ._llcrypt unregistered [ 6840.226226] Key type ._llcrypt registered [ 6840.230254] Key type .llcrypt registered [ 6840.369922] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 6859.564787] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 6878.390590] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 6879.826679] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6880.012566] alg: No test for adler32 (adler32-zlib) [ 6881.113798] Lustre: Lustre: Build Version: 2.17.52_126_g9629464 [ 6881.447809] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 6881.660111] Key type lgssc registered [ 6884.963892] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6930.876987] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6930.934910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6932.473137] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6932.559776] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6932.771985] Lustre: lustre-MDT0000: new disk, initializing [ 6932.915190] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6932.966581] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6939.162135] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6953.415974] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6953.572622] Lustre: 266934:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6953.665983] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6953.672274] Lustre: Skipped 1 previous similar message [ 6953.812265] Lustre: lustre-MDT0001: new disk, initializing [ 6953.987308] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6954.096082] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6954.112759] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6960.607151] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6969.848445] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6976.025114] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6984.489825] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6984.915494] Lustre: lustre-OST0000: new disk, initializing [ 6984.925893] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6985.102717] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6992.544723] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6992.571854] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6992.757931] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6993.610254] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 7002.481476] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7008.692929] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7008.976642] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7015.293370] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7015.700782] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7023.588135] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7023.589911] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7023.608890] LustreError: Skipped 1 previous similar message [ 7023.610884] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7023.680028] Lustre: Skipped 1 previous similar message [ 7028.713035] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7028.719800] Lustre: Skipped 2 previous similar messages [ 7033.312871] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7033.757666] Lustre: server umount lustre-OST0000 complete [ 7038.945531] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7038.975528] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7044.065425] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7044.080904] Lustre: Skipped 2 previous similar messages [ 7047.002845] Lustre: server umount lustre-MDT0000 complete [ 7052.416427] LustreError: 268678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777580180 with bad export cookie 3084883147204814728 [ 7052.422196] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7052.446324] LustreError: 268678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7053.318487] Lustre: server umount lustre-MDT0001 complete [ 7064.023518] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 16:16:30 (1777580190) [ 7071.526960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7072.007697] LustreError: 270948:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7072.030666] LustreError: 270948:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 7072.115921] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7077.234157] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 7077.345407] LustreError: 270949:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7082.482617] LustreError: 270948:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7087.596164] LustreError: 270949:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7088.678048] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7089.333642] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7094.913320] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 7103.245847] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7109.074674] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7118.798794] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7119.558380] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7127.360303] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 7134.887715] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7157.729162] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7157.741477] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7157.748689] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7157.759976] Lustre: Skipped 2 previous similar messages [ 7157.777844] Lustre: Skipped 1 previous similar message [ 7161.654522] Lustre: server umount lustre-OST0000 complete [ 7167.970018] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7167.983433] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7167.995665] Lustre: Skipped 1 previous similar message [ 7172.747591] Lustre: server umount lustre-MDT0000 complete [ 7173.099571] LustreError: 270943:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7173.129121] LustreError: 270943:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 7176.901675] LustreError: 270929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777580305 with bad export cookie 3084883147204815610 [ 7176.918791] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7176.923809] LustreError: 270929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7177.248988] Lustre: server umount lustre-MDT0001 complete [ 7192.749555] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 7197.174085] Key type lgssc unregistered [ 7197.631427] LNet: 274152:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7197.636870] LNetError: 274152:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7197.655514] LNet: Removed LNI 192.168.202.138@tcp [ 7198.868791] Key type .llcrypt unregistered [ 7198.874457] Key type ._llcrypt unregistered [ 7234.716759] Key type ._llcrypt registered [ 7234.726642] Key type .llcrypt registered [ 7234.813105] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 7253.877379] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 7255.141241] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7255.252482] alg: No test for adler32 (adler32-zlib) [ 7256.450839] Lustre: Lustre: Build Version: 2.17.52_126_g9629464 [ 7256.741769] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 7257.010227] Key type lgssc registered [ 7260.208204] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7311.952157] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7311.973231] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7313.330653] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7313.377304] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7313.448740] Lustre: lustre-MDT0000: new disk, initializing [ 7313.547636] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7313.577604] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7318.807302] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 7330.147596] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7330.296377] Lustre: 277950:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7330.422356] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7330.429518] Lustre: Skipped 1 previous similar message [ 7330.574429] Lustre: lustre-MDT0001: new disk, initializing [ 7330.725721] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7330.767872] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7330.794677] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7335.719818] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 7345.443187] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7351.182542] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7360.640075] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7361.104497] Lustre: lustre-OST2710: new disk, initializing [ 7361.112605] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 7361.276996] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 7369.427640] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 7369.447487] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 7369.619549] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 7370.089345] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 7380.839789] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 7389.743383] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7390.034716] Lustre: lustre-OST03e8: new disk, initializing [ 7390.037979] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 7390.139604] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 7397.569600] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 7398.065090] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 7398.082571] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 7398.217691] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 7404.545699] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 7414.184681] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 7414.705609] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7420.940525] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 7421.200087] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7426.530402] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 7426.853595] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7432.617817] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 7432.847604] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7438.308610] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7438.324548] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7438.344724] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7438.528773] Lustre: server umount lustre-MDT0000 complete [ 7439.337978] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7439.364484] Lustre: Skipped 2 previous similar messages [ 7442.673595] LustreError: 277942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777580571 with bad export cookie 1706949133167094347 [ 7442.685070] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7442.688914] LustreError: 277942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7443.037640] Lustre: server umount lustre-MDT0001 complete [ 7457.381558] Lustre: server umount lustre-OST2710 complete [ 7459.823149] Lustre: 275747:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777580572/real 1777580572] req@000000000056cdb4 x1863928129853568/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1777580588 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7459.869254] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7462.971525] Lustre: server umount lustre-OST03e8 complete [ 7490.799696] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 7505.134432] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 7546.459530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7546.810040] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7546.876553] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7547.016720] Lustre: lustre-MDT0000: new disk, initializing [ 7547.114289] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7547.137494] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7552.167471] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 7564.967340] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7565.110816] Lustre: 285703:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7565.173253] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7565.177438] Lustre: Skipped 1 previous similar message [ 7565.302605] Lustre: lustre-MDT0001: new disk, initializing [ 7565.476358] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7565.533139] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7565.549859] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7571.729934] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 7579.831783] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7583.920511] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7592.322120] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7592.654885] Lustre: lustre-OST0000: new disk, initializing [ 7592.658628] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7592.813743] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7599.255455] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 7601.260556] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7601.273909] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7601.441034] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7607.233984] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7611.752732] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7612.031740] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7617.233791] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7617.512713] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7622.114210] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7622.129103] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7622.147775] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7622.158236] Lustre: Skipped 1 previous similar message [ 7627.235401] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7632.362553] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7632.379344] Lustre: Skipped 1 previous similar message [ 7634.401807] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7634.807711] Lustre: server umount lustre-OST0000 complete [ 7642.081147] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7642.098979] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7642.122262] Lustre: Skipped 1 previous similar message [ 7642.147186] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7642.174663] Lustre: Skipped 1 previous similar message [ 7652.833741] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7652.846238] Lustre: Skipped 4 previous similar messages [ 7654.372724] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7654.591736] Lustre: server umount lustre-MDT0000 complete [ 7657.954158] LustreError: 285710:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7657.990621] LustreError: 285710:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 6 previous similar messages [ 7659.405885] LustreError: 285694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777580787 with bad export cookie 1706949133167096027 [ 7659.420752] LustreError: 285694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7659.421646] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7659.717530] Lustre: server umount lustre-MDT0001 complete [ 7671.096305] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 16:26:37 (1777580797) [ 7673.505250] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 7676.121327] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 16:26:42 (1777580802) [ 7680.765097] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 7714.662708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7714.965215] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7714.988923] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7715.215135] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7720.761944] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 7729.927142] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7730.073568] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7730.100293] Lustre: 291137:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7734.901673] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 7756.979837] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7757.269433] Lustre: Denying initial registration attempt from nid 192.168.202.138@tcp, specified as failover [ 7757.274487] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 7757.289416] LustreError: 292559:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -99 [ 7757.294898] LustreError: 292559:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 7757.299563] LustreError: 292559:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 7757.367421] Lustre: server umount lustre-OST0000 complete [ 7757.371127] LustreError: 292559:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 7765.998095] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7766.003225] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7766.026791] Lustre: Skipped 2 previous similar messages [ 7766.048954] Lustre: Skipped 1 previous similar message [ 7770.483603] Lustre: server umount lustre-MDT0000 complete [ 7771.118251] LustreError: 291146:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7771.153436] LustreError: 291146:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 7774.995670] LustreError: 291864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777580903 with bad export cookie 1706949133167096909 [ 7775.005922] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7775.006749] LustreError: 291864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7775.494438] Lustre: server umount lustre-MDT0001 complete [ 7787.719435] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 7791.227904] Key type lgssc unregistered [ 7791.609954] LNet: 293664:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7791.615433] LNetError: 293664:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7791.636605] LNet: Removed LNI 192.168.202.138@tcp [ 7792.682072] Key type .llcrypt unregistered [ 7792.683335] Key type ._llcrypt unregistered [ 7806.426299] Key type ._llcrypt registered [ 7806.436598] Key type .llcrypt registered [ 7806.591965] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 7808.197289] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7808.265188] alg: No test for adler32 (adler32-zlib) [ 7809.557528] Lustre: Lustre: Build Version: 2.17.52_126_g9629464 [ 7810.000087] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 7810.284335] Key type lgssc registered [ 7813.221609] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7863.592476] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 7872.778309] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7872.799711] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7874.186766] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7874.225400] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7874.575849] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7879.107163] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 7888.003628] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7888.123062] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7888.172338] Lustre: 296398:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7888.477860] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7893.647397] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 7899.923577] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7903.696747] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7923.973459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7924.312570] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7924.497386] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7931.619582] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 7939.155664] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7944.164262] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7944.170580] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7944.172707] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7944.185523] Lustre: Skipped 1 previous similar message [ 7944.232697] Lustre: Skipped 1 previous similar message [ 7947.836958] Lustre: server umount lustre-OST0000 complete [ 7954.404564] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7954.412777] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7954.425579] Lustre: Skipped 1 previous similar message [ 7954.431871] Lustre: Skipped 1 previous similar message [ 7958.489281] Lustre: server umount lustre-MDT0000 complete [ 7963.054807] LustreError: 296391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777581091 with bad export cookie 6282012131796059557 [ 7963.062710] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7963.077290] LustreError: 296391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7963.448760] Lustre: server umount lustre-MDT0001 complete [ 7975.302536] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 7978.402664] Key type lgssc unregistered [ 7978.651555] LNet: 299615:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7978.658804] LNetError: 299615:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7978.675465] LNet: Removed LNI 192.168.202.138@tcp [ 7979.621556] Key type .llcrypt unregistered [ 7979.623582] Key type ._llcrypt unregistered [ 8010.513911] Key type ._llcrypt registered [ 8010.516685] Key type .llcrypt registered [ 8010.643836] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 8012.192714] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8012.277321] alg: No test for adler32 (adler32-zlib) [ 8013.467879] Lustre: Lustre: Build Version: 2.17.52_126_g9629464 [ 8013.677214] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 8013.811271] Key type lgssc registered [ 8016.968692] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8025.816983] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8025.836624] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8027.398808] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8032.191805] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 8036.838492] LustreError: 301022:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8036.865740] LustreError: 301022:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 8041.956893] LustreError: 301021:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8042.266031] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8042.740842] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8047.510965] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 8053.191970] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8057.552886] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8067.967269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8068.435398] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8075.830662] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 8082.340884] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8097.253192] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8097.255439] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8097.278417] Lustre: Skipped 1 previous similar message [ 8097.295054] Lustre: Skipped 2 previous similar messages [ 8099.539595] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.38@tcp (stopping) [ 8102.251685] Lustre: server umount lustre-MDT0000 complete [ 8102.374931] LustreError: 301022:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8102.410083] LustreError: 301022:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 8104.622398] LustreError: 301018:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8106.506326] LustreError: 301003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777581234 with bad export cookie 11363100459742008269 [ 8106.518866] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8106.535705] LustreError: 301003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8107.506301] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8107.530322] Lustre: Skipped 1 previous similar message [ 8107.540069] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8109.771429] LustreError: 301018:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8109.790705] LustreError: 301018:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 8109.806402] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.38@tcp (stopping) [ 8113.057057] Lustre: server umount lustre-MDT0001 complete [ 8120.676401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8130.503242] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8131.808318] LustreError: 304281:0:(import.c:337:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 8131.826394] LustreError: 304281:0:(import.c:361:ptlrpc_invalidate_import()) @@@ still on sending list req@000000007870c86a x1863928923595392/t0(0) o250->MGC192.168.202.138@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1777581260 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8131.889205] LustreError: 304281:0:(import.c:371:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 8132.068610] LustreError: 300329:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000c12d613b x1863928923595776/t0(0) o250->MGC192.168.202.138@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8132.566357] LustreError: 302627:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8132.589053] LustreError: 302627:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 8132.684477] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8132.803368] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 8137.938821] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 8142.829631] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8148.714096] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8149.165561] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8153.314310] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 8154.611882] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8160.123874] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8162.490254] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8166.947111] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8168.959104] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8178.677341] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8178.695985] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8178.718754] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8178.729312] Lustre: Skipped 1 previous similar message [ 8179.692159] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8182.328250] Lustre: server umount lustre-OST0000 complete [ 8189.923260] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8189.931804] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8189.950091] Lustre: Skipped 1 previous similar message [ 8189.955795] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8189.957896] Lustre: Skipped 3 previous similar messages [ 8193.264697] Lustre: server umount lustre-MDT0000 complete [ 8195.553309] LustreError: 304294:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8195.564440] LustreError: 304294:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 9 previous similar messages [ 8197.974855] LustreError: 301002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777581326 with bad export cookie 11363100459742015920 [ 8197.989968] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8197.999640] LustreError: 301002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8198.673191] Lustre: server umount lustre-MDT0001 complete [ 8210.869461] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 8214.168348] Key type lgssc unregistered [ 8214.549800] LNet: 306727:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8214.560066] LNetError: 306727:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8214.575251] LNet: Removed LNI 192.168.202.138@tcp [ 8215.519877] Key type .llcrypt unregistered [ 8215.521625] Key type ._llcrypt unregistered [ 8242.252041] Key type ._llcrypt registered [ 8242.254597] Key type .llcrypt registered [ 8242.389972] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 8243.599205] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8243.701031] alg: No test for adler32 (adler32-zlib) [ 8244.832447] Lustre: Lustre: Build Version: 2.17.52_126_g9629464 [ 8245.119692] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 8245.291617] Key type lgssc registered [ 8248.555353] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8256.758633] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8256.779480] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8258.566108] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8263.288134] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 8267.758912] LustreError: 308128:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8267.779959] LustreError: 308128:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 8272.886284] LustreError: 308127:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8273.049187] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8273.442228] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8279.062909] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 8286.963439] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8291.198419] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8298.987504] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8299.000906] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8299.011700] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8299.017924] Lustre: Skipped 1 previous similar message [ 8300.070099] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8300.081458] Lustre: Skipped 2 previous similar messages [ 8300.634847] Lustre: server umount lustre-MDT0000 complete [ 8305.079592] LustreError: 308107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777581433 with bad export cookie 7208416349043868936 [ 8305.082807] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8305.099040] LustreError: 308107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8305.422996] Lustre: server umount lustre-MDT0001 complete [ 8325.666402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8326.083666] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8326.137324] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8326.414952] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8332.023155] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 8342.314556] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8342.439234] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8342.495346] Lustre: 310490:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8342.730319] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8347.686685] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 8354.909527] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8359.659534] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8371.533350] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8372.005680] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 8372.015566] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 8372.024603] LustreError: 312150:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 8372.034130] LustreError: 312150:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 8372.042250] LustreError: 312150:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 8372.148515] Lustre: server umount lustre-OST0000 complete [ 8372.154085] LustreError: 312150:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 8386.555061] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8386.918091] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8387.265425] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8394.012284] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 8397.501830] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 8400.023607] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8408.800363] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8408.936686] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 8409.010949] Lustre: lustre-OST0001: new disk, initializing [ 8409.014249] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8409.119841] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8415.183025] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 8418.964709] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8418.976589] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8419.120379] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8421.190970] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 8424.425718] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8424.437555] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8424.451963] LustreError: Skipped 1 previous similar message [ 8424.461483] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8424.477177] Lustre: Skipped 1 previous similar message [ 8424.497687] Lustre: Skipped 1 previous similar message [ 8426.252974] Lustre: server umount lustre-OST0001 complete [ 8429.541607] LustreError: 312618:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8433.132803] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8433.139652] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8433.159318] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8434.669751] LustreError: 312615:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8434.699655] LustreError: 312615:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 8436.600103] Lustre: server umount lustre-OST0000 complete [ 8444.897872] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8444.907290] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8444.910501] Lustre: Skipped 2 previous similar messages [ 8444.921158] Lustre: Skipped 4 previous similar messages [ 8447.125201] Lustre: server umount lustre-MDT0000 complete [ 8450.018101] LustreError: 310503:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8450.041886] LustreError: 310503:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 8450.735472] LustreError: 312632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777581579 with bad export cookie 7208416349043869489 [ 8450.748336] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8451.189713] Lustre: server umount lustre-MDT0001 complete [ 8463.206690] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 8466.355817] Key type lgssc unregistered [ 8466.788693] LNet: 315404:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8466.810413] LNetError: 315404:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8466.846385] LNet: Removed LNI 192.168.202.138@tcp [ 8467.838654] Key type .llcrypt unregistered [ 8467.842376] Key type ._llcrypt unregistered [ 8548.574232] Key type ._llcrypt registered [ 8548.589371] Key type .llcrypt registered [ 8548.699440] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 8566.168314] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 8568.048158] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8568.123535] alg: No test for adler32 (adler32-zlib) [ 8569.201260] Lustre: Lustre: Build Version: 2.17.52_126_g9629464 [ 8569.390630] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 8569.515587] Key type lgssc registered [ 8572.121609] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8610.117175] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8610.137364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8611.506549] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8611.562871] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8611.663367] Lustre: lustre-MDT0000: new disk, initializing [ 8611.796155] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8611.828485] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8616.818547] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 8629.017856] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8629.221206] Lustre: 320310:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8629.338766] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8629.343138] Lustre: Skipped 1 previous similar message [ 8629.398561] Lustre: lustre-MDT0001: new disk, initializing [ 8629.557528] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8629.605205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8629.624644] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8635.457890] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 8642.489548] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8645.871932] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8654.804974] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8655.089309] Lustre: lustre-OST0000: new disk, initializing [ 8655.092732] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8655.191807] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8661.778989] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 8664.726474] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8664.751686] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8664.966339] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8669.825524] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8674.602260] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8674.823232] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8679.288942] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8679.613505] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8685.541397] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8685.562457] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8685.566511] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8685.573444] Lustre: Skipped 1 previous similar message [ 8687.717415] Lustre: server umount lustre-OST0000 complete [ 8695.783325] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8695.784739] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8695.798267] Lustre: Skipped 2 previous similar messages [ 8695.804123] Lustre: Skipped 1 previous similar message [ 8698.018370] Lustre: server umount lustre-MDT0000 complete [ 8702.233662] LustreError: 320302:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777581830 with bad export cookie 14535940460324814580 [ 8702.236426] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8702.249791] LustreError: 320302:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8702.689899] Lustre: server umount lustre-MDT0001 complete [ 8711.658055] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 16:43:58 (1777581838) [ 8720.044491] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 16:44:06 (1777581846) [ 8730.546940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8731.443429] LustreError: 324660:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8731.474367] LustreError: 324660:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 8731.556917] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8736.749731] LustreError: 324661:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8737.184295] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 8740.840457] LustreError: 324660:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8745.956165] LustreError: 324661:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8747.165628] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8747.769333] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8752.778137] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 8758.675927] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8763.044961] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8771.026714] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8771.417171] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8777.976928] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 8784.600096] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8793.814399] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 8796.887386] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 8799.816607] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 8802.956398] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 8805.798083] Lustre: Failing over lustre-MDT0000 [ 8806.212584] Lustre: server umount lustre-MDT0000 complete [ 8807.397354] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8807.401586] LustreError: 324656:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8807.429596] Lustre: Skipped 2 previous similar messages [ 8807.481145] LustreError: 324656:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 8817.635447] LustreError: 324657:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8817.664892] LustreError: 324657:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 7 previous similar messages [ 8822.753120] Lustre: 318533:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777581935/real 1777581935] req@00000000834abee9 x1863929505584384/t0(0) o400->MGC192.168.202.138@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1777581951 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8822.785958] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8827.237987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8833.378221] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8833.450896] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 8833.709760] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 8838.641403] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8838.676940] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 8838.760541] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 8839.052832] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 8848.965666] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8851.055769] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8856.461358] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 8859.297073] Lustre: Failing over lustre-MDT0000 [ 8859.371856] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.38@tcp (stopping) [ 8860.165750] Lustre: server umount lustre-MDT0000 complete [ 8864.230916] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8864.233028] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8864.250617] LustreError: 326810:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8864.251720] Lustre: Skipped 2 previous similar messages [ 8864.284657] LustreError: 326810:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 15 previous similar messages [ 8880.608255] Lustre: 318533:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777581992/real 1777581992] req@00000000e6f6a622 x1863929505607040/t0(0) o400->MGC192.168.202.138@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1777582008 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8880.677856] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8882.314549] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8891.421798] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8891.520309] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 8895.160282] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 8896.537224] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8896.558179] Lustre: Skipped 2 previous similar messages [ 8896.607103] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 8896.663099] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 8897.881202] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 8906.324956] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8908.026231] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8916.961271] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8916.981975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8916.982649] Lustre: Skipped 1 previous similar message [ 8917.004607] Lustre: Skipped 2 previous similar messages [ 8921.425133] Lustre: server umount lustre-MDT0000 complete [ 8922.086615] LustreError: 324655:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8922.105385] LustreError: 324655:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 25 previous similar messages [ 8925.596647] LustreError: 324642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777582053 with bad export cookie 14535940460324817296 [ 8925.601165] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8925.607391] LustreError: 324642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8926.097565] Lustre: server umount lustre-MDT0001 complete [ 8941.737267] Lustre: server umount lustre-OST0000 complete [ 8961.506533] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8962.256058] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8968.604359] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 8979.437994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8986.886195] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 8992.537653] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8997.611416] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9005.736366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9007.242285] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 9013.044565] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 9021.604759] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9036.266481] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9036.284890] Lustre: Skipped 1 previous similar message [ 9036.306906] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9037.174986] Lustre: server umount lustre-OST0000 complete [ 9046.498247] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9046.501254] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9046.502123] Lustre: Skipped 2 previous similar messages [ 9056.224809] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9056.501831] Lustre: server umount lustre-MDT0000 complete [ 9060.841977] LustreError: 330442:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9060.870905] LustreError: 330442:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 5 previous similar messages [ 9061.860648] LustreError: 330423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777582190 with bad export cookie 14535940460324818038 [ 9061.865893] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9061.876232] LustreError: 330423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9062.828886] Lustre: server umount lustre-MDT0001 complete [ 9078.129904] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 9081.510642] Key type lgssc unregistered [ 9081.912033] LNet: 333641:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9081.923651] LNetError: 333641:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9081.947473] LNet: Removed LNI 192.168.202.138@tcp [ 9083.080995] Key type .llcrypt unregistered [ 9083.083908] Key type ._llcrypt unregistered [ 9116.740562] Key type ._llcrypt registered [ 9116.746133] Key type .llcrypt registered [ 9116.972136] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 9118.548789] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9118.705695] alg: No test for adler32 (adler32-zlib) [ 9119.947131] Lustre: Lustre: Build Version: 2.17.52_126_g9629464 [ 9120.398403] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 9120.657822] Key type lgssc registered [ 9123.663405] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9134.084786] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9134.104078] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9136.009680] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9141.383108] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 9146.353835] LustreError: 335145:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9146.383395] LustreError: 335145:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 9151.461657] LustreError: 335144:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9152.495353] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9153.125394] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9158.973862] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 9167.294960] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9172.012047] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9182.132290] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9182.615675] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9189.466831] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 9195.417737] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9206.152327] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 9219.553466] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9219.560951] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9219.600367] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9221.604058] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9221.606540] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9221.618374] Lustre: Skipped 1 previous similar message [ 9221.636386] Lustre: Skipped 1 previous similar message [ 9223.305580] Lustre: server umount lustre-MDT0000 complete [ 9226.741857] LustreError: 335144:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9226.767609] LustreError: 335144:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 9227.346607] LustreError: 335124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777582355 with bad export cookie 15983553045519899129 [ 9227.357291] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9227.814529] Lustre: server umount lustre-MDT0001 complete [ 9242.215864] Lustre: server umount lustre-OST0000 complete [ 9263.938372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9264.894373] LustreError: 338540:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9265.166211] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9270.246915] LustreError: 338541:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9270.256420] LustreError: 338541:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 9272.659541] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 9280.494735] LustreError: 338541:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9280.505865] LustreError: 338541:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 9284.412598] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9285.161633] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9290.745695] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 9298.528628] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9303.531930] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9312.419711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9313.090651] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9320.525731] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 9327.169558] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9336.668347] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 9351.652344] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9351.672473] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9351.702968] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9351.713364] Lustre: Skipped 1 previous similar message [ 9357.284532] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9357.312207] Lustre: Skipped 5 previous similar messages [ 9357.429906] Lustre: server umount lustre-MDT0000 complete [ 9361.392201] LustreError: 338520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777582489 with bad export cookie 15983553045519900242 [ 9361.408543] LustreError: 338520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9361.409549] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9361.710272] Lustre: server umount lustre-MDT0001 complete [ 9376.601179] Lustre: server umount lustre-OST0000 complete [ 9402.517594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9403.499687] LustreError: 342135:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9403.644114] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9409.720118] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 9420.115810] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9425.455529] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 9432.847698] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9437.261696] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9446.929215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9447.512422] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9447.515397] Lustre: Skipped 1 previous similar message [ 9448.575202] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 9455.723306] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 9461.918793] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9469.409498] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9469.421966] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9469.441918] Lustre: Skipped 2 previous similar messages [ 9469.457188] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9472.903810] Lustre: server umount lustre-OST0000 complete [ 9477.091090] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9477.097549] LustreError: Skipped 1 previous similar message [ 9477.099764] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9477.122036] Lustre: Skipped 1 previous similar message [ 9480.673417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9480.676424] Lustre: Skipped 3 previous similar messages [ 9483.209980] Lustre: server umount lustre-MDT0000 complete [ 9485.799079] LustreError: 342135:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9485.816603] LustreError: 342135:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 4 previous similar messages [ 9487.525960] LustreError: 342116:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777582615 with bad export cookie 15983553045519901264 [ 9487.544298] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9487.549343] LustreError: 342116:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 9488.285360] Lustre: server umount lustre-MDT0001 complete [ 9502.185839] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 9505.483633] Key type lgssc unregistered [ 9505.841928] LNet: 345313:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9505.856175] LNetError: 345313:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9505.878430] LNet: Removed LNI 192.168.202.138@tcp [ 9506.758457] Key type .llcrypt unregistered [ 9506.761477] Key type ._llcrypt unregistered [ 9536.796572] Key type ._llcrypt registered [ 9536.798666] Key type .llcrypt registered [ 9536.887534] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 9537.848173] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9537.956855] alg: No test for adler32 (adler32-zlib) [ 9539.200426] Lustre: Lustre: Build Version: 2.17.52_126_g9629464 [ 9539.532517] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 9539.715037] Key type lgssc registered [ 9542.550698] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9551.344364] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9551.361864] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9551.367382] LustreError: 346741:0:(osd_handler.c:8530:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 9551.385863] LustreError: 346741:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 9551.391718] LustreError: 346741:0:(obd_mount.c:250:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 9551.400809] LustreError: 346741:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 9551.415945] LustreError: 346741:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 9561.294492] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9561.310119] LustreError: 347138:0:(osd_handler.c:8530:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 9561.337974] LustreError: 347138:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 9561.344323] LustreError: 347138:0:(obd_mount.c:250:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 9561.354941] LustreError: 347138:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 9561.380423] LustreError: 347138:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 9583.648144] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 9586.255377] Key type lgssc unregistered [ 9586.489933] LNet: 348124:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9586.502938] LNetError: 348124:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9586.516734] LNet: Removed LNI 192.168.202.138@tcp [ 9587.428608] Key type .llcrypt unregistered [ 9587.430505] Key type ._llcrypt unregistered [ 9607.262250] Key type ._llcrypt registered [ 9607.264652] Key type .llcrypt registered [ 9607.391498] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 9622.875150] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 9623.782529] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9623.826569] alg: No test for adler32 (adler32-zlib) [ 9624.865308] Lustre: Lustre: Build Version: 2.17.52_126_g9629464 [ 9625.078483] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 9625.191111] Key type lgssc registered [ 9627.739673] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9665.909297] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9665.972932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9667.243519] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9667.273816] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9667.342782] Lustre: lustre-MDT0000: new disk, initializing [ 9667.451625] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9667.477344] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9673.147243] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 9685.366101] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9685.490232] Lustre: 351250:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9685.533465] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9685.537745] Lustre: Skipped 1 previous similar message [ 9685.592257] Lustre: lustre-MDT0001: new disk, initializing [ 9685.676182] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9685.709984] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9685.722989] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9690.110417] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 9698.302433] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9701.700278] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9709.408859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9709.700551] Lustre: lustre-OST0000: new disk, initializing [ 9709.717923] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9709.832627] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9716.640250] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 9717.884275] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9717.900953] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9718.080972] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9726.293858] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9731.672354] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9731.967336] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9738.200865] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9738.467656] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9743.843493] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9743.846187] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9743.879506] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9743.895984] Lustre: Skipped 1 previous similar message [ 9746.953547] Lustre: server umount lustre-OST0000 complete [ 9752.545457] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9752.549242] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9752.557103] Lustre: Skipped 1 previous similar message [ 9752.567680] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9754.089500] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9754.091904] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9754.117869] Lustre: Skipped 1 previous similar message [ 9759.205310] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9759.222767] Lustre: Skipped 1 previous similar message [ 9764.321837] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9764.325840] Lustre: Skipped 1 previous similar message [ 9766.369930] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9766.613706] Lustre: server umount lustre-MDT0000 complete [ 9770.634684] LustreError: 353003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777582899 with bad export cookie 7178699844510390811 [ 9770.636947] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9770.652413] LustreError: 353003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9771.115965] Lustre: server umount lustre-MDT0001 complete [ 9782.154813] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 9784.110892] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 17:01:50 (1777582910) [ 9792.012635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9792.833038] LustreError: 355319:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9792.860834] LustreError: 355319:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 9792.956540] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9798.119635] LustreError: 355320:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9798.958803] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 9803.262085] LustreError: 355319:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9808.354594] LustreError: 355320:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9810.224328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9810.753780] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9815.498834] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 9821.617679] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9825.696598] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9834.045665] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9834.535568] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9841.821612] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 9847.342841] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9854.744991] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9854.898945] Lustre: lustre-OST0001: new disk, initializing [ 9854.902756] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9855.059768] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9861.262490] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9861.294086] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9861.378965] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9862.313886] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 9872.101515] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 9880.340734] Lustre: lustre-OST0001: Not available for connect from 192.168.202.38@tcp (stopping) [ 9881.572336] Lustre: server umount lustre-OST0001 complete [ 9882.099154] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9882.103588] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9882.116543] LustreError: Skipped 1 previous similar message [ 9882.117735] LustreError: 356917:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9882.214124] Lustre: Skipped 1 previous similar message [ 9892.333548] LustreError: 358120:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9892.363536] LustreError: 358120:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 5 previous similar messages [ 9905.123756] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9905.132042] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9905.167498] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9909.353726] Lustre: server umount lustre-OST0000 complete [ 9917.922398] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9917.946208] Lustre: Skipped 2 previous similar messages [ 9928.160178] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9928.393341] Lustre: server umount lustre-MDT0000 complete [ 9932.544429] LustreError: 355299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777583060 with bad export cookie 7178699844510391693 [ 9932.546055] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9932.569713] LustreError: 355299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 9939.203521] Lustre: server umount lustre-MDT0001 complete [ 9952.501058] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 9955.663969] Key type lgssc unregistered [ 9955.993875] LNet: 359674:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9956.005550] LNetError: 359674:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9956.034093] LNet: Removed LNI 192.168.202.138@tcp [ 9956.974641] Key type .llcrypt unregistered [ 9956.978817] Key type ._llcrypt unregistered [10012.935660] Key type ._llcrypt registered [10012.941827] Key type .llcrypt registered [10013.079404] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [10014.580308] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10014.677653] alg: No test for adler32 (adler32-zlib) [10015.771228] Lustre: Lustre: Build Version: 2.17.52_126_g9629464 [10016.048753] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [10016.204495] Key type lgssc registered [10018.931686] Lustre: Echo OBD driver; http://www.lustre.org/ [10026.716731] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10026.743791] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10028.158362] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10028.202148] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10028.410729] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10033.486756] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [10042.913558] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10043.015497] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10043.043044] Lustre: 362461:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10043.357941] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10048.310203] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [10054.081956] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10057.698935] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10060.769051] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10060.772965] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10060.797819] Lustre: Skipped 1 previous similar message [10065.894902] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10065.908247] Lustre: Skipped 2 previous similar messages [10066.423902] Lustre: server umount lustre-MDT0000 complete [10070.239432] LustreError: 362452:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777583198 with bad export cookie 12961371615803909803 [10070.250211] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10070.257534] LustreError: 362452:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10070.809677] Lustre: server umount lustre-MDT0001 complete [10079.826747] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 17:06:46 (1777583206) [10086.878527] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10087.460568] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10092.523738] LustreError: 364746:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10092.551088] LustreError: 364746:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [10093.519913] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [10097.640146] LustreError: 364745:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10102.517451] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10102.910783] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10108.051088] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [10114.334980] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10118.341665] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10126.805256] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10127.047642] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10127.442150] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10135.063245] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [10142.572178] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10152.092939] Lustre: Permanently deactivating lustre-OST0000 [10152.108516] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [10158.183132] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10161.869914] LustreError: 367346:0:(mgs_llog.c:2031:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [10161.883478] LustreError: 367346:0:(mgs_llog.c:2086:mgs_replace_nids()) Only MGS is allowed to be started [10161.899263] LustreError: 367346:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10166.276543] LustreError: 367394:0:(mgs_llog.c:2086:mgs_replace_nids()) Only MGS is allowed to be started [10166.280495] LustreError: 367394:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10174.973294] Lustre: server umount lustre-OST0000 complete [10177.461820] LustreError: 367644:0:(mgs_llog.c:2086:mgs_replace_nids()) Only MGS is allowed to be started [10177.469711] LustreError: 367644:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10182.627780] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10182.642120] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10182.645566] Lustre: Skipped 1 previous similar message [10182.672237] Lustre: Skipped 1 previous similar message [10187.745373] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10187.771158] Lustre: Skipped 1 previous similar message [10192.866665] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10192.882781] Lustre: Skipped 1 previous similar message [10194.400177] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10194.591704] Lustre: server umount lustre-MDT0000 complete [10197.988376] LustreError: 364745:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10198.033846] LustreError: 364745:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [10199.232664] LustreError: 366352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777583327 with bad export cookie 12961371615803910384 [10199.235762] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10199.253104] LustreError: 366352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10208.983358] Lustre: server umount lustre-MDT0001 complete [10219.497613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10223.760040] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [10226.964965] LustreError: 368787:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [10234.159730] Lustre: 369126:0:(mgs_llog.c:1798:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.138@tcp [10236.985625] Lustre: server umount MGS complete [10249.778434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10250.336412] LustreError: 369849:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10250.350355] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10250.362025] Lustre: Skipped 1 previous similar message [10250.420106] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10254.330066] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [10255.844741] LustreError: 369850:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10263.406788] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10263.835631] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10268.393840] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [10273.633594] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10277.632819] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10279.899857] Lustre: Permanently reactivating lustre-OST0000 [10279.913383] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [10279.918908] Lustre: Skipped 2 previous similar messages [10308.071751] LustreError: 371402:0:(osp_dev.c:695:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [10317.789861] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10318.151882] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10318.161258] Lustre: Skipped 1 previous similar message [10325.066895] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [10332.503120] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10343.907056] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10343.910347] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10343.921334] LustreError: Skipped 1 previous similar message [10343.924175] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10343.952042] Lustre: Skipped 1 previous similar message [10346.006149] Lustre: server umount lustre-OST0000 complete [10354.149236] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10354.182257] Lustre: Skipped 1 previous similar message [10358.117837] Lustre: server umount lustre-MDT0000 complete [10359.265983] LustreError: 369849:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10359.278873] LustreError: 369849:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [10363.427546] LustreError: 372834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777583491 with bad export cookie 12961371615803911434 [10363.440303] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10363.443074] LustreError: 372834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10364.019136] Lustre: server umount lustre-MDT0001 complete [10376.068217] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [10378.758748] Key type lgssc unregistered [10378.997130] LNet: 373859:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10379.004598] LNetError: 373859:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10380.071656] LNet: Removed LNI 192.168.202.138@tcp [10381.172213] Key type .llcrypt unregistered [10381.174099] Key type ._llcrypt unregistered [10402.338734] Key type ._llcrypt registered [10402.342875] Key type .llcrypt registered [10402.469855] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [10423.591864] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [10425.324033] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10425.701264] alg: No test for adler32 (adler32-zlib) [10426.860829] Lustre: Lustre: Build Version: 2.17.52_126_g9629464 [10427.186824] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [10427.462475] Key type lgssc registered [10430.769746] Lustre: Echo OBD driver; http://www.lustre.org/ [10471.989116] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10472.058071] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10473.373712] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10473.420326] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10473.534243] Lustre: lustre-MDT0000: new disk, initializing [10473.629240] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10473.648546] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10478.663889] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [10488.892427] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10489.045151] Lustre: 376983:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10489.097632] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10489.101836] Lustre: Skipped 1 previous similar message [10489.194133] Lustre: lustre-MDT0001: new disk, initializing [10489.295484] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10489.322053] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10489.341229] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10493.847846] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [10502.434557] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10506.773100] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10515.149996] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10515.504411] Lustre: lustre-OST0000: new disk, initializing [10515.507797] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10515.611377] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10521.701988] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10521.720364] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10521.872578] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10523.532409] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [10531.861676] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10537.914389] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10538.243431] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10545.510804] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10545.946411] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10552.803078] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10552.806085] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10552.828736] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10552.831588] Lustre: Skipped 1 previous similar message [10552.852292] Lustre: Skipped 1 previous similar message [10555.630478] Lustre: server umount lustre-OST0000 complete [10561.025865] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10561.033385] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10561.050639] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10563.044392] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10563.045029] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10568.166156] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10568.175454] Lustre: Skipped 2 previous similar messages [10573.283820] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10573.304221] Lustre: Skipped 1 previous similar message [10575.328752] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10575.625655] Lustre: server umount lustre-MDT0000 complete [10579.881929] LustreError: 376975:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777583708 with bad export cookie 16790646980307623886 [10579.896592] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10579.899656] LustreError: 376975:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10580.389852] Lustre: server umount lustre-MDT0001 complete [10591.645693] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [10594.099418] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 17:15:20 (1777583720) [10605.161030] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10606.120602] LustreError: 381040:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10606.147934] LustreError: 381040:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [10606.257524] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10611.693141] LustreError: 381041:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10613.775484] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [10615.796669] LustreError: 381040:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10620.910083] LustreError: 381041:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10626.032406] LustreError: 381040:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10626.301602] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10626.686607] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10632.144704] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [10638.707477] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10643.416269] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10651.022173] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10651.231973] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10658.338841] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [10666.303140] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10669.439401] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [10673.725257] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [10683.363566] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10683.373043] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10683.391095] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10683.405399] Lustre: Skipped 1 previous similar message [10683.424076] Lustre: Skipped 1 previous similar message [10686.190632] Lustre: server umount lustre-OST0000 complete [10693.098991] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10693.111135] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10702.818041] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10702.829679] Lustre: Skipped 5 previous similar messages [10705.380341] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10705.691979] Lustre: server umount lustre-MDT0000 complete [10707.944960] LustreError: 383389:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10707.960852] LustreError: 383389:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [10709.621036] LustreError: 381020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777583837 with bad export cookie 16790646980307624768 [10709.640787] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10710.022483] Lustre: server umount lustre-MDT0001 complete [10721.276368] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [10724.179404] Key type lgssc unregistered [10724.509390] LNet: 384413:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10724.519400] LNetError: 384413:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10724.530243] LNet: Removed LNI 192.168.202.138@tcp [10725.451386] Key type .llcrypt unregistered [10725.456720] Key type ._llcrypt unregistered [10751.060113] Key type ._llcrypt registered [10751.078179] Key type .llcrypt registered [10751.239254] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [10751.631880] Key type .llcrypt unregistered [10751.640799] Key type ._llcrypt unregistered [10774.517495] Key type ._llcrypt registered [10774.519952] Key type .llcrypt registered [10774.639745] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [10776.912841] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10777.013478] alg: No test for adler32 (adler32-zlib) [10778.336985] Lustre: Lustre: Build Version: 2.17.52_126_g9629464 [10778.642076] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [10778.846372] Key type lgssc registered [10782.371218] Lustre: Echo OBD driver; http://www.lustre.org/ [10791.699901] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10791.758493] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10793.445063] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10798.693341] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [10802.669787] LustreError: 386600:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10802.708249] LustreError: 386600:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [10807.780538] LustreError: 386599:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10808.328414] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10808.752854] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10813.922024] LustreError: 386600:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10813.951388] LustreError: 386600:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [10815.848902] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [10818.033813] LustreError: 386599:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10818.058494] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10818.071962] LustreError: 386599:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [10821.901540] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10823.158048] LustreError: 386600:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10823.199486] LustreError: 386600:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [10829.591462] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10829.919423] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10835.504620] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [10848.742604] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10848.761594] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10848.798265] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10850.276520] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10850.291560] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10850.291842] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10850.346465] Lustre: Skipped 1 previous similar message [10852.314654] Lustre: server umount lustre-OST0000 complete [10860.515914] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10860.523475] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10860.532327] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10860.547445] Lustre: Skipped 1 previous similar message [10860.568267] Lustre: Skipped 1 previous similar message [10862.917118] Lustre: server umount lustre-MDT0000 complete [10865.636492] LustreError: 386600:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10865.668107] LustreError: 386600:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [10867.170837] LustreError: 386581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777583995 with bad export cookie 13553894766744632332 [10867.180536] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10867.186336] LustreError: 386581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10867.563458] Lustre: server umount lustre-MDT0001 complete [10881.658132] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [10885.273565] Key type lgssc unregistered [10885.658770] LNet: 389584:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10885.663090] LNetError: 389584:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10885.716917] LNet: Removed LNI 192.168.202.138@tcp [10886.881375] Key type .llcrypt unregistered [10886.884103] Key type ._llcrypt unregistered [10920.044449] Key type ._llcrypt registered [10920.055386] Key type .llcrypt registered [10920.314346] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [10922.700187] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10922.834188] alg: No test for adler32 (adler32-zlib) [10924.532815] Lustre: Lustre: Build Version: 2.17.52_126_g9629464 [10925.021144] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [10925.286450] Key type lgssc registered [10928.526877] Lustre: Echo OBD driver; http://www.lustre.org/ [10937.917838] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10937.940400] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10953.504534] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [10958.625112] LustreError: 390964:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.138@tcp: failed processing log, type 4: rc = -110 [10985.382343] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10993.767227] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [11002.369856] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11011.516479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11012.328291] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11012.454172] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11017.698243] LustreError: 390989:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11017.718144] LustreError: 390989:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [11018.107095] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [11019.770376] LustreError: 390990:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11019.815410] LustreError: 390990:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [11024.873184] LustreError: 390989:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11024.895979] LustreError: 390989:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [11029.286603] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11029.675990] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11035.679982] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [11043.857488] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11048.034434] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11056.296197] Lustre: server umount lustre-OST0000 complete [11060.193040] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11060.203567] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11060.225374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11060.705193] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11060.729217] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11065.825883] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11065.837754] Lustre: Skipped 1 previous similar message [11070.205352] Lustre: server umount lustre-MDT0000 complete [11070.947333] LustreError: 391892:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11070.983065] LustreError: 391892:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [11075.842520] LustreError: 390972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777584204 with bad export cookie 17667709014529884293 [11075.848118] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11075.855440] LustreError: 390972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11086.648839] Lustre: server umount lustre-MDT0001 complete [11101.725380] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [11105.001494] Key type lgssc unregistered [11105.517269] LNet: 394178:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11105.523526] LNetError: 394178:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11105.548247] LNet: Removed LNI 192.168.202.138@tcp [11106.634802] Key type .llcrypt unregistered [11106.636602] Key type ._llcrypt unregistered [11135.947765] Key type ._llcrypt registered [11135.950562] Key type .llcrypt registered [11136.097996] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [11137.763691] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11137.979664] alg: No test for adler32 (adler32-zlib) [11139.136342] Lustre: Lustre: Build Version: 2.17.52_126_g9629464 [11139.378029] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [11139.513320] Key type lgssc registered [11142.699734] Lustre: Echo OBD driver; http://www.lustre.org/ [11154.447964] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11154.498269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11156.281421] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11162.861451] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [11165.665961] LustreError: 395582:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11165.696330] LustreError: 395582:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [11170.785928] LustreError: 395581:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11172.648359] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11173.213706] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11179.190615] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [11187.056178] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11192.556153] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11202.324202] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11202.873446] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11210.282414] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [11217.592253] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11226.416322] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11234.789389] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11234.800257] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11234.827188] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11236.839664] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11236.842252] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11236.860964] Lustre: Skipped 1 previous similar message [11236.881869] Lustre: Skipped 3 previous similar messages [11239.261989] Lustre: server umount lustre-MDT0000 complete [11240.111181] LustreError: 395576:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11241.382522] LustreError: 395562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.38@tcp arrived at 1777584369 with bad export cookie 17345595649045495176 [11243.342913] LustreError: 395561:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777584371 with bad export cookie 17345595649045494511 [11243.355646] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11243.357481] LustreError: 395561:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11244.003794] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11244.030492] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11244.074251] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11247.073554] LustreError: 396293:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11247.073801] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11247.087871] LustreError: 396293:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 4 previous similar messages [11249.637935] Lustre: server umount lustre-OST0000 complete [11253.226763] LustreError: 395577:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11253.246898] LustreError: 395577:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [11256.936598] Lustre: server umount lustre-MDT0001 complete [11269.093812] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [11272.187453] Key type lgssc unregistered [11272.521358] LNet: 398935:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11272.539398] LNetError: 398935:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11273.578930] LNet: Removed LNI 192.168.202.138@tcp [11274.610482] Key type .llcrypt unregistered [11274.616163] Key type ._llcrypt unregistered [11305.997173] Key type ._llcrypt registered [11306.000817] Key type .llcrypt registered [11306.132457] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [11308.383301] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11308.402579] alg: No test for adler32 (adler32-zlib) [11309.571521] Lustre: Lustre: Build Version: 2.17.52_126_g9629464 [11309.859812] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [11310.056678] Key type lgssc registered [11313.380706] Lustre: Echo OBD driver; http://www.lustre.org/ [11324.183929] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11324.222608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11326.391422] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11332.081538] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [11335.668559] LustreError: 400340:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11335.693105] LustreError: 400340:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [11340.779794] LustreError: 400341:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11342.594446] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11343.317122] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11349.805683] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [11355.707137] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11360.573776] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11368.471726] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11368.964040] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11371.020054] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11376.793884] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [11383.609731] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11389.430381] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [11389.446761] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11389.489693] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11391.459076] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11391.463699] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11394.836332] Lustre: server umount lustre-MDT0001 complete [11396.607100] LustreError: 402033:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11396.643779] LustreError: 402033:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [11401.712992] LustreError: 400340:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11401.736949] LustreError: 400340:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [11406.818502] LustreError: 400341:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11406.818892] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11406.823502] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11406.824986] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11406.825098] Lustre: Skipped 2 previous similar messages [11408.353517] Lustre: server umount lustre-OST0000 complete [11414.696344] Lustre: server umount lustre-MDT0000 complete [11431.115797] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [11434.371797] Key type lgssc unregistered [11434.620258] LNet: 403669:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11434.635168] LNetError: 403669:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11434.661886] LNet: Removed LNI 192.168.202.138@tcp [11435.806179] Key type .llcrypt unregistered [11435.808034] Key type ._llcrypt unregistered [11468.294659] Key type ._llcrypt registered [11468.301630] Key type .llcrypt registered [11468.468110] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [11484.183779] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [11485.363873] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11485.448085] alg: No test for adler32 (adler32-zlib) [11486.978807] Lustre: Lustre: Build Version: 2.17.52_126_g9629464 [11487.174698] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [11487.303476] Key type lgssc registered [11490.089767] Lustre: Echo OBD driver; http://www.lustre.org/ [11540.706661] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [11564.795046] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11564.833717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11566.204678] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11566.241892] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11566.360831] Lustre: lustre-MDT0000: new disk, initializing [11566.464619] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11566.502367] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11572.978626] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [11592.232466] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11592.654220] Lustre: lustre-OST0000: new disk, initializing [11592.658882] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11592.663665] Lustre: Skipped 1 previous similar message [11592.745485] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11600.188658] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [11601.091359] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [11601.115539] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [11601.194317] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [11609.987937] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11628.332121] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11628.467448] Lustre: 407639:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11628.570930] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11628.718445] Lustre: lustre-MDT0001: new disk, initializing [11628.865228] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11628.908339] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [11628.927622] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [11634.236603] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [11645.700787] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [11645.902556] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [11650.996364] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [11651.158385] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [11660.513492] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.38@tcp (stopping) [11660.789983] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11660.794115] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [11660.801124] Lustre: Skipped 1 previous similar message [11662.669576] Lustre: server umount lustre-MDT0001 complete [11671.013831] LustreError: 407646:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11671.032845] LustreError: 407646:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 5 previous similar messages [11676.136425] LustreError: 409145:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11681.250678] LustreError: 407651:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11681.253030] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11681.290454] LustreError: 407651:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [11681.304249] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11681.324387] Lustre: Skipped 2 previous similar messages [11686.369339] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11686.370920] LustreError: 407650:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11686.858679] Lustre: server umount lustre-OST0000 complete [11698.091875] Lustre: server umount lustre-MDT0000 complete [11713.243756] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [11716.406653] Key type lgssc unregistered [11716.759486] LNet: 412213:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11716.799123] LNetError: 412213:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11716.826660] LNet: Removed LNI 192.168.202.138@tcp [11717.809285] Key type .llcrypt unregistered [11717.810919] Key type ._llcrypt unregistered [11764.070168] Key type ._llcrypt registered [11764.071841] Key type .llcrypt registered [11764.266836] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [11764.734618] Key type .llcrypt unregistered [11764.744521] Key type ._llcrypt unregistered [11784.853360] Key type ._llcrypt registered [11784.859810] Key type .llcrypt registered [11785.016918] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [11786.121967] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11786.269559] alg: No test for adler32 (adler32-zlib) [11787.273944] Lustre: Lustre: Build Version: 2.17.52_126_g9629464 [11787.574284] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [11787.727914] Key type lgssc registered [11790.509932] Lustre: Echo OBD driver; http://www.lustre.org/ [11825.848789] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11825.894913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11827.358705] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11827.424515] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11827.587308] Lustre: lustre-MDT0000: new disk, initializing [11827.752969] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11827.789126] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11832.653831] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [11845.271501] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11845.429920] Lustre: 415559:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11845.516625] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11845.526837] Lustre: Skipped 1 previous similar message [11845.664821] Lustre: lustre-MDT0001: new disk, initializing [11845.831720] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11845.893667] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11845.914083] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11850.880062] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [11859.844848] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11864.873382] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11873.709780] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11874.243155] Lustre: lustre-OST0000: new disk, initializing [11874.248713] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11874.374421] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11881.631611] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [11883.123225] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11883.132749] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11883.330886] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11890.654897] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11898.852649] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11898.858026] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11898.881241] Lustre: Skipped 2 previous similar messages [11898.903851] Lustre: Skipped 2 previous similar messages [11903.979131] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11903.987451] Lustre: Skipped 2 previous similar messages [11904.815832] Lustre: server umount lustre-MDT0000 complete [11908.554135] LustreError: 415551:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777585036 with bad export cookie 7447272225328357993 [11908.556257] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11908.562732] LustreError: 415551:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11908.883387] Lustre: server umount lustre-MDT0001 complete [11922.923523] Lustre: server umount lustre-OST0000 complete [11935.866548] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 17:37:42 (1777585062) [11945.922056] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11946.865665] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11952.113529] LustreError: 419219:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11952.139872] LustreError: 419219:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [11952.158944] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [11957.217527] LustreError: 419218:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11962.114539] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11962.349771] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [11962.610420] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11966.796826] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [11974.547231] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11980.370720] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12003.826878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12004.076161] Lustre: Found index 0 for lustre-OST0000, updating log [12004.242615] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12011.718163] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [12018.303867] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12018.572326] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [12023.790119] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12023.790335] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12023.811632] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12023.825429] Lustre: Skipped 1 previous similar message [12025.537739] Lustre: server umount lustre-OST0000 complete [12041.187917] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12041.191550] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12041.198198] Lustre: Skipped 2 previous similar messages [12041.216335] Lustre: Skipped 3 previous similar messages [12047.511429] Lustre: server umount lustre-MDT0000 complete [12051.428815] LustreError: 419943:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12051.455176] LustreError: 419943:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [12051.560614] LustreError: 419198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777585179 with bad export cookie 7447272225328359386 [12051.565473] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12051.580967] LustreError: 419198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12051.997645] Lustre: server umount lustre-MDT0001 complete [12060.893275] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 17:39:47 (1777585187) [12063.072130] LNet: 1 local NIs in recovery (showing 1): 192.168.202.138@tcp [12685.844168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12686.283494] LustreError: 425290:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12686.401167] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12690.779241] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [12691.451157] LustreError: 425291:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12696.547123] LustreError: 425290:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12698.597610] LustreError: 425290:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12698.626759] LustreError: 425290:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [12700.798943] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12701.074548] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12706.426158] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [12713.845188] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12717.090175] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12738.032566] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12738.377566] Lustre: Found index 0 for lustre-OST0000, updating log [12738.632444] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12741.987023] LNet: 1 local NIs in recovery (showing 1): 192.168.202.138@tcp [12746.161839] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [12751.126590] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12753.325370] Lustre: server umount lustre-OST0000 complete [12757.476810] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12757.491261] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12757.512582] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12757.518697] Lustre: Skipped 2 previous similar messages [12758.563130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12758.563804] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12760.692442] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12760.712504] Lustre: Skipped 4 previous similar messages [12763.704590] Lustre: server umount lustre-MDT0000 complete [12763.752448] LustreError: 426882:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12763.789197] LustreError: 426882:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [12767.359461] LustreError: 425272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777585895 with bad export cookie 7447272225328360471 [12767.372057] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12767.375717] LustreError: 425272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12767.634794] Lustre: server umount lustre-MDT0001 complete [12797.297727] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [12802.400378] LNet: 1 local NIs in recovery (showing 1): 192.168.202.138@tcp [12808.641498] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [12841.739589] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 17:52:48 (1777585968) [12859.293711] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [12862.210230] Key type lgssc unregistered [12862.434089] LNet: 431507:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12862.440737] LNetError: 431507:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12990.438782] LNet: Removed LNI 192.168.202.138@tcp [12991.395597] Key type .llcrypt unregistered [12991.398480] Key type ._llcrypt unregistered [13007.343955] Key type ._llcrypt registered [13007.351408] Key type .llcrypt registered [13007.458813] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [13008.554882] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13008.664458] alg: No test for adler32 (adler32-zlib) [13009.659907] Lustre: Lustre: Build Version: 2.17.52_126_g9629464 [13009.859018] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [13009.993762] Key type lgssc registered [13012.723709] Lustre: Echo OBD driver; http://www.lustre.org/ [13016.486484] LNet: Added LNI 192.168.202.138@tcp100 [8/256/0/180] [13017.350890] LNet: Added LNI 192.168.202.138@tcp101 [8/256/0/180] [13018.124423] LNet: Added LNI 192.168.202.138@tcp102 [8/256/0/180] [13018.933857] LNet: Added LNI 192.168.202.138@tcp103 [8/256/0/180] [13019.781570] LNet: Added LNI 192.168.202.138@tcp104 [8/256/0/180] [13020.660448] LNet: Added LNI 192.168.202.138@tcp105 [8/256/0/180] [13021.652503] LNet: Added LNI 192.168.202.138@tcp106 [8/256/0/180] [13022.457653] LNet: Added LNI 192.168.202.138@tcp107 [8/256/0/180] [13023.317077] LNet: Added LNI 192.168.202.138@tcp108 [8/256/0/180] [13024.009624] LNet: Added LNI 192.168.202.138@tcp109 [8/256/0/180] [13024.870217] LNet: Added LNI 192.168.202.138@tcp110 [8/256/0/180] [13025.710932] LNet: Added LNI 192.168.202.138@tcp111 [8/256/0/180] [13026.519744] LNet: Added LNI 192.168.202.138@tcp112 [8/256/0/180] [13027.394861] LNet: Added LNI 192.168.202.138@tcp113 [8/256/0/180] [13028.132576] LNet: Added LNI 192.168.202.138@tcp114 [8/256/0/180] [13028.809885] LNet: Added LNI 192.168.202.138@tcp115 [8/256/0/180] [13029.566745] LNet: Added LNI 192.168.202.138@tcp116 [8/256/0/180] [13030.330651] LNet: Added LNI 192.168.202.138@tcp117 [8/256/0/180] [13031.038720] LNet: Added LNI 192.168.202.138@tcp118 [8/256/0/180] [13031.772460] LNet: Added LNI 192.168.202.138@tcp119 [8/256/0/180] [13032.558748] LNet: Added LNI 192.168.202.138@tcp120 [8/256/0/180] [13033.251665] LNet: Added LNI 192.168.202.138@tcp121 [8/256/0/180] [13034.051048] LNet: Added LNI 192.168.202.138@tcp122 [8/256/0/180] [13034.871514] LNet: Added LNI 192.168.202.138@tcp123 [8/256/0/180] [13035.680669] LNet: Added LNI 192.168.202.138@tcp124 [8/256/0/180] [13036.424510] LNet: Added LNI 192.168.202.138@tcp125 [8/256/0/180] [13037.197121] LNet: Added LNI 192.168.202.138@tcp126 [8/256/0/180] [13038.130918] LNet: Added LNI 192.168.202.138@tcp127 [8/256/0/180] [13038.975663] LNet: Added LNI 192.168.202.138@tcp128 [8/256/0/180] [13039.686217] LNet: Added LNI 192.168.202.138@tcp129 [8/256/0/180] [13040.467495] LNet: Added LNI 192.168.202.138@tcp130 [8/256/0/180] [13041.258784] LNet: Added LNI 192.168.202.138@tcp131 [8/256/0/180] [13041.948070] LNet: Added LNI 192.168.202.138@tcp132 [8/256/0/180] [13042.608222] LNet: Added LNI 192.168.202.138@tcp133 [8/256/0/180] [13043.336437] LNet: Added LNI 192.168.202.138@tcp134 [8/256/0/180] [13044.179597] LNet: Added LNI 192.168.202.138@tcp135 [8/256/0/180] [13050.114716] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13050.148328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13051.278732] Lustre: 434525:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.138@tcp: can't add NID 192.168.202.138@tcp130: rc = -75 [13051.424906] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13051.446969] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13051.513490] Lustre: lustre-MDT0000: new disk, initializing [13051.595297] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13051.612809] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13055.906469] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [13066.670853] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13066.912414] Lustre: 434541:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13066.982649] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13066.988403] Lustre: Skipped 1 previous similar message [13067.080197] Lustre: lustre-MDT0001: new disk, initializing [13067.108681] Lustre: 435323:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.138@tcp: can't add NID 192.168.202.138@tcp130: rc = -75 [13067.118281] Lustre: 435323:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [13067.221870] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13067.264787] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13067.279935] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13071.491635] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [13075.029460] Lustre: 435692:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.138@tcp: can't add NID 192.168.202.138@tcp130: rc = -75 [13075.039891] Lustre: 435692:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [13078.817971] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13082.193632] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13088.392524] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13088.617974] Lustre: lustre-OST0000: new disk, initializing [13088.621793] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13088.707904] Lustre: 436292:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.138@tcp: can't add NID 192.168.202.138@tcp130: rc = -75 [13088.714394] Lustre: 436292:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [13088.763299] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13094.970331] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [13098.524935] Lustre: 436746:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.138@tcp: can't add NID 192.168.202.138@tcp130: rc = -75 [13098.536317] Lustre: 436746:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [13098.586501] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13098.601714] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13098.783396] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13102.614700] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13109.217984] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13109.219994] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13109.231055] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13109.237553] Lustre: Skipped 1 previous similar message [13114.345089] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13114.365080] Lustre: Skipped 2 previous similar messages [13114.759705] Lustre: server umount lustre-OST0000 complete [13118.434358] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13118.449873] LustreError: Skipped 1 previous similar message [13118.457253] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13118.464386] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13119.457425] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13124.579727] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13124.585856] Lustre: Skipped 3 previous similar messages [13129.698712] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13129.707224] Lustre: Skipped 1 previous similar message [13132.256190] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13132.538298] Lustre: server umount lustre-MDT0000 complete [13135.298577] LustreError: 434535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777586263 with bad export cookie 12305326232963843912 [13135.309338] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13135.313454] LustreError: 434535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13135.649740] Lustre: server umount lustre-MDT0001 complete [13144.710701] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [13147.196327] Key type lgssc unregistered [13147.557938] LNet: 438070:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13147.568165] LNetError: 438070:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13147.577996] LNet: Removed LNI 192.168.202.138@tcp [13148.640260] LNet: Removed LNI 192.168.202.138@tcp100 [13148.665705] LNetError: 438070:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13148.674313] LNetError: 438070:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13148.688126] LNet: Removed LNI 192.168.202.138@tcp101 [13149.728370] LNet: Removed LNI 192.168.202.138@tcp102 [13149.741897] LNetError: 438070:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13149.755041] LNetError: 438070:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13150.816278] LNet: Removed LNI 192.168.202.138@tcp103 [13151.840218] LNet: Removed LNI 192.168.202.138@tcp104 [13151.844367] LNetError: 438070:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13151.853444] LNetError: 438070:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13151.867252] LNet: Removed LNI 192.168.202.138@tcp105 [13152.928714] LNet: Removed LNI 192.168.202.138@tcp106 [13153.952196] LNet: Removed LNI 192.168.202.138@tcp107 [13154.976215] LNet: Removed LNI 192.168.202.138@tcp108 [13156.001060] LNet: Removed LNI 192.168.202.138@tcp109 [13156.013422] LNetError: 438070:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13156.024146] LNetError: 438070:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [13156.042629] LNet: Removed LNI 192.168.202.138@tcp110 [13157.088582] LNet: Removed LNI 192.168.202.138@tcp111 [13158.113545] LNet: Removed LNI 192.168.202.138@tcp112 [13158.115566] LNet: Removed LNI 192.168.202.138@tcp113 [13158.126364] LNet: Removed LNI 192.168.202.138@tcp114 [13158.130323] LNet: Removed LNI 192.168.202.138@tcp115 [13158.136629] LNet: Removed LNI 192.168.202.138@tcp116 [13158.140484] LNet: Removed LNI 192.168.202.138@tcp117 [13158.144543] LNet: Removed LNI 192.168.202.138@tcp118 [13158.147651] LNet: Removed LNI 192.168.202.138@tcp119 [13158.160982] LNet: Removed LNI 192.168.202.138@tcp120 [13158.177510] LNet: Removed LNI 192.168.202.138@tcp121 [13158.179970] LNet: Removed LNI 192.168.202.138@tcp122 [13158.188188] LNet: Removed LNI 192.168.202.138@tcp123 [13158.191807] LNet: Removed LNI 192.168.202.138@tcp124 [13158.194937] LNet: Removed LNI 192.168.202.138@tcp125 [13158.202060] LNet: Removed LNI 192.168.202.138@tcp126 [13158.203946] LNet: Removed LNI 192.168.202.138@tcp127 [13158.207505] LNet: Removed LNI 192.168.202.138@tcp128 [13158.210579] LNet: Removed LNI 192.168.202.138@tcp129 [13158.217154] LNet: Removed LNI 192.168.202.138@tcp130 [13158.222243] LNet: Removed LNI 192.168.202.138@tcp131 [13158.230036] LNet: Removed LNI 192.168.202.138@tcp132 [13158.237699] LNet: Removed LNI 192.168.202.138@tcp133 [13158.244456] LNet: Removed LNI 192.168.202.138@tcp134 [13158.262814] LNet: Removed LNI 192.168.202.138@tcp135 [13159.064883] Key type .llcrypt unregistered [13159.067181] Key type ._llcrypt unregistered [13180.592965] Key type ._llcrypt registered [13180.594867] Key type .llcrypt registered [13180.686861] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [13193.466026] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [13194.992083] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13195.036750] alg: No test for adler32 (adler32-zlib) [13196.118917] Lustre: Lustre: Build Version: 2.17.52_126_g9629464 [13196.335278] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [13196.459419] Key type lgssc registered [13198.929594] Lustre: Echo OBD driver; http://www.lustre.org/ [13228.353652] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 17:59:15 (1777586355) [13246.594788] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [13249.736288] Key type lgssc unregistered [13250.046244] LNet: 442083:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13250.051240] LNetError: 442083:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13250.065239] LNet: Removed LNI 192.168.202.138@tcp [13250.819483] Key type .llcrypt unregistered [13250.821403] Key type ._llcrypt unregistered [13265.872588] Key type ._llcrypt registered [13265.875444] Key type .llcrypt registered [13265.964812] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [13267.208862] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13267.235646] alg: No test for adler32 (adler32-zlib) [13268.262165] Lustre: Lustre: Build Version: 2.17.52_126_g9629464 [13268.421134] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [13268.577034] Key type lgssc registered [13271.136927] Lustre: Echo OBD driver; http://www.lustre.org/ [13279.155809] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13279.204375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13280.427963] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13280.453452] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13280.517582] Lustre: lustre-MDT0000: new disk, initializing [13280.583334] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13280.605703] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13284.091301] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [13293.414567] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13293.520301] Lustre: 443280:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13293.575140] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13293.578051] Lustre: Skipped 1 previous similar message [13293.624109] Lustre: lustre-MDT0001: new disk, initializing [13293.714830] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13293.752533] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13293.766780] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13297.223752] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [13303.915236] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13307.114515] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13313.611246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13313.868373] Lustre: lustre-OST0000: new disk, initializing [13313.872212] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13313.958303] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13319.375692] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [13323.864387] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13323.878400] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13324.016104] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13326.467822] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13330.014600] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing lnet_if_list [13335.183358] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [13335.211540] LNet: Added LNI 192.168.202.138@tcp10 [8/256/0/180] [13339.789771] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [13339.819500] LNet: Added LNI 192.168.202.138@tcp11 [8/256/0/180] [13343.999449] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [13344.031793] LNet: Added LNI 192.168.202.138@tcp12 [8/256/0/180] [13347.999941] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [13348.025659] LNet: Added LNI 192.168.202.138@tcp13 [8/256/0/180] [13352.075378] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [13352.102539] LNet: Added LNI 192.168.202.138@tcp14 [8/256/0/180] [13356.212394] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [13356.256571] LNet: Added LNI 192.168.202.138@tcp15 [8/256/0/180] [13361.036285] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [13361.069369] LNet: Added LNI 192.168.202.138@tcp16 [8/256/0/180] [13365.265777] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [13365.288029] LNet: Added LNI 192.168.202.138@tcp17 [8/256/0/180] [13369.426624] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [13369.455715] LNet: Added LNI 192.168.202.138@tcp18 [8/256/0/180] [13374.343549] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [13374.404616] LNet: Added LNI 192.168.202.138@tcp19 [8/256/0/180] [13379.902902] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [13379.929327] LNet: Added LNI 192.168.202.138@tcp20 [8/256/0/180] [13384.656628] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [13384.682333] LNet: Added LNI 192.168.202.138@tcp21 [8/256/0/180] [13390.127109] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [13390.155855] LNet: Added LNI 192.168.202.138@tcp22 [8/256/0/180] [13395.287334] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [13395.316745] LNet: Added LNI 192.168.202.138@tcp23 [8/256/0/180] [13400.656611] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [13400.689036] LNet: Added LNI 192.168.202.138@tcp24 [8/256/0/180] [13407.249182] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [13407.307206] LNet: Added LNI 192.168.202.138@tcp25 [8/256/0/180] [13412.422846] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [13412.493774] LNet: Added LNI 192.168.202.138@tcp26 [8/256/0/180] [13418.226617] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [13418.251709] LNet: Added LNI 192.168.202.138@tcp27 [8/256/0/180] [13424.072436] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [13424.098445] LNet: Added LNI 192.168.202.138@tcp28 [8/256/0/180] [13429.660745] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [13429.718412] LNet: Added LNI 192.168.202.138@tcp29 [8/256/0/180] [13435.156896] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [13435.188467] LNet: Added LNI 192.168.202.138@tcp30 [8/256/0/180] [13441.238829] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [13441.266296] LNet: Added LNI 192.168.202.138@tcp31 [8/256/0/180] [13447.460823] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [13447.502600] LNet: Added LNI 192.168.202.138@tcp32 [8/256/0/180] [13454.856481] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [13454.889213] LNet: Added LNI 192.168.202.138@tcp33 [8/256/0/180] [13460.785398] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [13460.812574] LNet: Added LNI 192.168.202.138@tcp34 [8/256/0/180] [13466.496443] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [13466.517823] LNet: Added LNI 192.168.202.138@tcp35 [8/256/0/180] [13472.606747] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [13472.632162] LNet: Added LNI 192.168.202.138@tcp36 [8/256/0/180] [13479.411288] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [13479.433047] LNet: Added LNI 192.168.202.138@tcp37 [8/256/0/180] [13485.465272] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [13485.491558] LNet: Added LNI 192.168.202.138@tcp38 [8/256/0/180] [13491.517507] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [13491.550926] LNet: Added LNI 192.168.202.138@tcp39 [8/256/0/180] [13497.528621] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [13497.551654] LNet: Added LNI 192.168.202.138@tcp40 [8/256/0/180] [13504.195324] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [13504.225892] LNet: Added LNI 192.168.202.138@tcp41 [8/256/0/180] [13511.943991] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [13511.983109] LNet: Added LNI 192.168.202.138@tcp42 [8/256/0/180] [13518.977795] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [13519.023023] LNet: Added LNI 192.168.202.138@tcp43 [8/256/0/180] [13526.022194] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [13526.064765] LNet: Added LNI 192.168.202.138@tcp44 [8/256/0/180] [13532.919461] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [13532.949810] LNet: Added LNI 192.168.202.138@tcp45 [8/256/0/180] [13540.041623] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [13540.074949] LNet: Added LNI 192.168.202.138@tcp46 [8/256/0/180] [13546.883208] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [13546.918471] LNet: Added LNI 192.168.202.138@tcp47 [8/256/0/180] [13553.736543] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [13553.754922] LNet: Added LNI 192.168.202.138@tcp48 [8/256/0/180] [13560.231435] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [13560.262918] LNet: Added LNI 192.168.202.138@tcp49 [8/256/0/180] [13566.988343] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [13567.010635] LNet: Added LNI 192.168.202.138@tcp50 [8/256/0/180] [13585.382067] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13585.390086] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13585.397578] Lustre: Skipped 2 previous similar messages [13587.714426] Lustre: server umount lustre-MDT0000 complete [13591.334848] LustreError: 443271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777586719 with bad export cookie 14871976589039254494 [13591.343233] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13591.348067] LustreError: 443271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13591.673521] Lustre: server umount lustre-MDT0001 complete [13606.262849] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [13610.978408] Lustre: 442552:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777586723/real 1777586723] req@000000007e8d7ec6 x1863934432927104/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1777586739 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13610.992157] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13612.509673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13613.026318] Lustre: 442550:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777586725/real 1777586725] req@00000000272d41c4 x1863934432927232/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1777586741 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13616.160691] LustreError: 442548:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000007e8d7ec6 x1863934432928128/t0(0) o250->MGC192.168.202.138@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13616.424279] LustreError: 445012:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13616.439385] LustreError: 445012:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 4 previous similar messages [13616.490597] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13617.504118] Lustre: 442549:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777586729/real 1777586729] req@0000000027f341ab x1863934432927360/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1777586745 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13619.545546] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [13620.520575] LustreError: 445013:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13621.540120] LustreError: 445012:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13621.551719] LustreError: 445012:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [13622.562278] Lustre: 442552:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777586734/real 1777586734] req@00000000f82a8cc9 x1863934432927616/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1777586750 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13626.298310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13626.607362] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13627.872465] Lustre: 442551:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777586739/real 1777586739] req@000000004ef38e58 x1863934432927744/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1777586755 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13630.294740] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [13631.987237] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13634.943519] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13637.797857] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13640.173571] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13652.451121] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13652.456175] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13652.463852] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13652.471422] Lustre: Skipped 3 previous similar messages [13655.536697] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13655.541870] Lustre: Skipped 1 previous similar message [13658.532194] Lustre: server umount lustre-OST0000 complete [13662.691375] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13662.696204] LustreError: Skipped 1 previous similar message [13662.704677] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13662.718270] Lustre: Skipped 1 previous similar message [13662.723121] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13668.212744] Lustre: server umount lustre-MDT0000 complete [13670.882842] LustreError: 468399:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13670.882842] LustreError: 468403:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13671.459711] LustreError: 443272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777586799 with bad export cookie 14871976589039257049 [13671.459966] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13671.477414] LustreError: 443272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13671.728532] Lustre: server umount lustre-MDT0001 complete [13681.091660] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [13683.771562] Key type lgssc unregistered [13684.120573] LNet: 470934:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13684.135151] LNetError: 470934:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13684.143760] LNet: Removed LNI 192.168.202.138@tcp [13684.145792] LNet: Removed LNI 192.168.202.138@tcp10 [13684.158755] LNet: Removed LNI 192.168.202.138@tcp11 [13684.164720] LNet: Removed LNI 192.168.202.138@tcp12 [13684.170394] LNet: Removed LNI 192.168.202.138@tcp13 [13684.173478] LNet: Removed LNI 192.168.202.138@tcp14 [13684.178217] LNet: Removed LNI 192.168.202.138@tcp15 [13684.182441] LNet: Removed LNI 192.168.202.138@tcp16 [13684.188350] LNet: Removed LNI 192.168.202.138@tcp17 [13684.191904] LNet: Removed LNI 192.168.202.138@tcp18 [13684.197423] LNet: Removed LNI 192.168.202.138@tcp19 [13684.204352] LNet: Removed LNI 192.168.202.138@tcp20 [13684.211541] LNet: Removed LNI 192.168.202.138@tcp21 [13684.214942] LNet: Removed LNI 192.168.202.138@tcp22 [13684.220968] LNet: Removed LNI 192.168.202.138@tcp23 [13684.230411] LNet: Removed LNI 192.168.202.138@tcp24 [13684.237066] LNet: Removed LNI 192.168.202.138@tcp25 [13684.243060] LNet: Removed LNI 192.168.202.138@tcp26 [13684.245184] LNet: Removed LNI 192.168.202.138@tcp27 [13684.256659] LNet: Removed LNI 192.168.202.138@tcp28 [13684.265157] LNet: Removed LNI 192.168.202.138@tcp29 [13684.269957] LNet: Removed LNI 192.168.202.138@tcp30 [13684.279250] LNet: Removed LNI 192.168.202.138@tcp31 [13684.290458] LNet: Removed LNI 192.168.202.138@tcp32 [13684.298316] LNet: Removed LNI 192.168.202.138@tcp33 [13684.301550] LNet: Removed LNI 192.168.202.138@tcp34 [13684.304628] LNet: Removed LNI 192.168.202.138@tcp35 [13684.308414] LNet: Removed LNI 192.168.202.138@tcp36 [13684.311448] LNet: Removed LNI 192.168.202.138@tcp37 [13684.313524] LNet: Removed LNI 192.168.202.138@tcp38 [13684.316785] LNet: Removed LNI 192.168.202.138@tcp39 [13684.320725] LNet: Removed LNI 192.168.202.138@tcp40 [13684.325606] LNet: Removed LNI 192.168.202.138@tcp41 [13684.329397] LNet: Removed LNI 192.168.202.138@tcp42 [13684.332359] LNet: Removed LNI 192.168.202.138@tcp43 [13684.334673] LNet: Removed LNI 192.168.202.138@tcp44 [13684.336883] LNet: Removed LNI 192.168.202.138@tcp45 [13684.339647] LNet: Removed LNI 192.168.202.138@tcp46 [13684.341457] LNet: Removed LNI 192.168.202.138@tcp47 [13684.343213] LNet: Removed LNI 192.168.202.138@tcp48 [13684.344988] LNet: Removed LNI 192.168.202.138@tcp49 [13684.356712] LNet: Removed LNI 192.168.202.138@tcp50 [13685.153847] Key type .llcrypt unregistered [13685.155621] Key type ._llcrypt unregistered [13703.610635] Key type ._llcrypt registered [13703.612412] Key type .llcrypt registered [13703.716628] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [13716.627427] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [13717.588143] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13717.653129] alg: No test for adler32 (adler32-zlib) [13718.703836] Lustre: Lustre: Build Version: 2.17.52_126_g9629464 [13718.926451] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [13719.079928] Key type lgssc registered [13721.556939] Lustre: Echo OBD driver; http://www.lustre.org/ [13753.794167] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13753.819624] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13755.085654] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13755.112855] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13755.167331] Lustre: lustre-MDT0000: new disk, initializing [13755.241855] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13755.267560] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13758.673180] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [13767.501541] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13767.601352] Lustre: 474101:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13767.639361] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13767.642591] Lustre: Skipped 1 previous similar message [13767.700095] Lustre: lustre-MDT0001: new disk, initializing [13767.793864] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13767.831195] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13767.839885] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13771.332808] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [13777.915446] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13781.363816] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13788.535598] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13788.753876] Lustre: *** cfs_fail_loc=90f, val=0*** [13788.831379] Lustre: lustre-OST0000: new disk, initializing [13788.843493] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13788.943874] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13794.421884] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [13795.410202] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13795.423346] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13795.534364] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13800.887675] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13811.170183] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13811.177783] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13811.178302] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13811.197141] Lustre: Skipped 1 previous similar message [13811.250106] Lustre: Skipped 1 previous similar message [13814.569870] Lustre: server umount lustre-OST0000 complete [13818.856686] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13818.865855] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13818.878378] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13821.410980] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13821.422714] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13821.432070] Lustre: Skipped 1 previous similar message [13823.788239] Lustre: server umount lustre-MDT0000 complete [13827.143542] LustreError: 474091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777586955 with bad export cookie 448888239787866534 [13827.159558] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13827.164464] LustreError: 474091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13827.625037] Lustre: server umount lustre-MDT0001 complete [13835.789452] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [13837.931154] Key type lgssc unregistered [13838.142787] LNet: 477663:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13838.151761] LNetError: 477663:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13838.166181] LNet: Removed LNI 192.168.202.138@tcp [13838.772262] Key type .llcrypt unregistered [13838.774789] Key type ._llcrypt unregistered [13854.624165] Key type ._llcrypt registered [13854.626418] Key type .llcrypt registered [13854.713921] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [13855.413101] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13855.470080] alg: No test for adler32 (adler32-zlib) [13856.450169] Lustre: Lustre: Build Version: 2.17.52_126_g9629464 [13856.638127] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [13856.760976] Key type lgssc registered [13859.324109] Lustre: Echo OBD driver; http://www.lustre.org/ [13865.560475] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13865.580529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13866.905354] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13870.628541] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [13877.218350] LustreError: 478896:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13877.233723] LustreError: 478896:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [13877.956363] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13878.208336] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13881.505916] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [13885.890786] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13888.529805] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13893.806587] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13894.006688] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13899.910041] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [13904.532231] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13921.506633] loop0: detected capacity change from 0 to 204800000 [13921.620686] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13921.632920] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13921.640441] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13926.347271] loop0: detected capacity change from 0 to 204800000 [13926.453443] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13926.459177] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13926.468118] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13930.143214] loop0: detected capacity change from 0 to 204800000 [13934.769397] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13934.884532] Lustre: fs15246-OST0000: new disk, initializing [13934.889462] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [13934.950895] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13939.503574] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [13946.635182] loop1: detected capacity change from 0 to 204800000 [13950.950613] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13951.036321] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [13951.091580] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [13951.168261] Lustre: fs15246-MDT0000: new disk, initializing [13951.330865] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13951.352395] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13951.434639] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [13952.482382] Lustre: 483271:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@0000000010ea63a2 x1863935049376128/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1777587096 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [13954.653846] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [13961.572547] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13961.576607] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13961.595928] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13970.413585] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13970.426161] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [13974.750346] Lustre: server umount fs15246-MDT0000 complete [13975.532188] LustreError: 478891:0:(ldlm_lib.c:1180:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13980.646968] LustreError: 478892:0:(ldlm_lib.c:1180:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13981.553532] Lustre: server umount fs15246-OST0000 complete [13989.349386] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13989.353399] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13989.363843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13990.370040] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13990.377851] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13990.383180] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13995.435242] Lustre: server umount lustre-OST0000 complete [14001.123038] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14001.127130] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14001.133624] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14001.133633] Lustre: Skipped 1 previous similar message [14001.138324] Lustre: Skipped 1 previous similar message [14006.241841] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14006.245495] Lustre: Skipped 2 previous similar messages [14012.384134] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14012.503146] Lustre: server umount lustre-MDT0000 complete [14014.807869] LustreError: 478875:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777587143 with bad export cookie 10091018541473287192 [14014.809474] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14014.814076] LustreError: 478875:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14015.027989] Lustre: server umount lustre-MDT0001 complete [14022.348543] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [14024.309986] Key type lgssc unregistered [14024.523109] LNet: 486336:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14024.526935] LNetError: 486336:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14024.540807] LNet: Removed LNI 192.168.202.138@tcp [14025.113146] Key type .llcrypt unregistered [14025.115541] Key type ._llcrypt unregistered [14037.733841] Key type ._llcrypt registered [14037.735937] Key type .llcrypt registered [14037.803440] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [14038.293953] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14038.391837] alg: No test for adler32 (adler32-zlib) [14039.354371] Lustre: Lustre: Build Version: 2.17.52_126_g9629464 [14039.512721] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [14039.601289] Key type lgssc registered [14041.901356] Lustre: Echo OBD driver; http://www.lustre.org/ [14069.776624] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [14076.043365] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [14099.848310] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14099.875550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14101.079852] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14101.112204] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14101.218253] Lustre: lustre-MDT0000: new disk, initializing [14101.301415] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14101.317726] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14104.280403] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [14112.092824] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14112.171763] Lustre: 490742:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14112.224382] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14112.230765] Lustre: Skipped 1 previous similar message [14112.292619] Lustre: lustre-MDT0001: new disk, initializing [14112.396045] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14112.435675] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14112.441156] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14115.412561] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [14121.685132] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14125.177477] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14131.364414] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14131.568683] Lustre: lustre-OST0000: new disk, initializing [14131.572044] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14131.637488] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14136.222688] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [14137.905252] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14137.915893] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14137.996422] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14142.340959] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14145.290896] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [14145.464783] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [14148.747348] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [14148.895416] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [14153.702426] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14153.703979] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14153.710375] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14153.718269] Lustre: Skipped 1 previous similar message [14156.384377] Lustre: server umount lustre-OST0000 complete [14163.425274] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14163.433100] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14163.441813] Lustre: Skipped 1 previous similar message [14163.451973] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14165.462623] Lustre: server umount lustre-MDT0000 complete [14168.186277] LustreError: 490735:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777587296 with bad export cookie 3699267119478227766 [14168.188818] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14168.195705] LustreError: 490735:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14168.451420] Lustre: server umount lustre-MDT0001 complete [14185.779778] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [14193.357551] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [14219.733577] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14077 sec ========= 18:15:46 (1777587346) [14221.076313] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 18:15:48 (1777587348) === [14223.833585] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 18:15:50 (1777587350) === [14255.633477] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [14258.299162] Key type lgssc unregistered [14258.566877] LNet: 498923:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14258.571518] LNetError: 498923:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14258.592728] LNet: Removed LNI 192.168.202.138@tcp [14259.182361] Key type .llcrypt unregistered [14259.184532] Key type ._llcrypt unregistered