[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 446952900 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.996 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.003118] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.007706] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229833f6470, max_idle_ns: 440795327230 ns [ 0.008022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399996) [ 0.009021] pid_max: default: 32768 minimum: 301 [ 0.010129] LSM: Security Framework initializing [ 0.011051] Yama: becoming mindful. [ 0.012031] SELinux: Initializing. [ 0.013050] *** VALIDATE selinux *** [ 0.020757] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025138] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026151] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027303] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028107] *** VALIDATE tmpfs *** [ 0.030076] *** VALIDATE proc *** [ 0.031155] *** VALIDATE cgroup *** [ 0.032006] *** VALIDATE cgroup2 *** [ 0.033303] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035056] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037022] Spectre V2 : User space: Vulnerable [ 0.038005] Speculative Store Bypass: Vulnerable [ 0.041343] debug: unmapping init [mem 0xffffffff9d659000-0xffffffff9d660fff] [ 0.043123] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044682] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045016] ... version: 2 [ 0.045885] ... bit width: 48 [ 0.046007] ... generic registers: 4 [ 0.046752] ... value mask: 0000ffffffffffff [ 0.047008] ... max period: 00007fffffffffff [ 0.048008] ... fixed-purpose events: 3 [ 0.048806] ... event mask: 000000070000000f [ 0.049249] rcu: Hierarchical SRCU implementation. [ 0.051336] smp: Bringing up secondary CPUs ... [ 0.052597] x86: Booting SMP configuration: [ 0.053020] .... node #0, CPUs: #1 #2 #3 [ 0.056270] smp: Brought up 1 node, 4 CPUs [ 0.058008] smpboot: Max logical packages: 1 [ 0.059016] smpboot: Total of 4 processors activated (19199.96 BogoMIPS) [ 0.143931] node 0 deferred pages initialised in 82ms [ 0.147060] devtmpfs: initialized [ 0.148218] x86/mm: Memory block size: 128MB [ 0.150883] gcov: version magic: 0x41383552 [ 0.153175] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.156088] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.158343] pinctrl core: initialized pinctrl subsystem [ 0.160126] [ 0.160634] ************************************************************* [ 0.163010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.165009] ** ** [ 0.167010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.169010] ** ** [ 0.170006] ** This means that this kernel is built to expose internal ** [ 0.172006] ** IOMMU data structures, which may compromise security on ** [ 0.174023] ** your system. ** [ 0.176007] ** ** [ 0.177005] ** If you see this message and you are not debugging the ** [ 0.179007] ** kernel, report this immediately to your vendor! ** [ 0.181015] ** ** [ 0.185021] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.188023] ************************************************************* [ 0.191853] NET: Registered protocol family 16 [ 0.194451] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.197048] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.199052] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.202104] cpuidle: using governor menu [ 0.203519] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.205844] PCI: Using configuration type 1 for base access [ 0.209096] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.216138] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.219055] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.222070] cryptd: max_cpu_qlen set to 1000 [ 0.224213] ACPI: Added _OSI(Module Device) [ 0.225018] ACPI: Added _OSI(Processor Device) [ 0.227012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.229006] ACPI: Added _OSI(Processor Aggregator Device) [ 0.233000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.240561] ACPI: Interpreter enabled [ 0.242076] ACPI: PM: (supports S0 S3 S4 S5) [ 0.243015] ACPI: Using IOAPIC for interrupt routing [ 0.244121] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.248397] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.256765] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.259025] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.262020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.265092] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.270529] acpiphp: Slot [2] registered [ 0.272070] acpiphp: Slot [5] registered [ 0.273085] acpiphp: Slot [6] registered [ 0.275082] acpiphp: Slot [7] registered [ 0.276101] acpiphp: Slot [8] registered [ 0.278161] acpiphp: Slot [9] registered [ 0.279133] acpiphp: Slot [10] registered [ 0.281088] acpiphp: Slot [3] registered [ 0.282176] acpiphp: Slot [4] registered [ 0.284085] acpiphp: Slot [11] registered [ 0.285052] acpiphp: Slot [12] registered [ 0.287115] acpiphp: Slot [13] registered [ 0.289104] acpiphp: Slot [14] registered [ 0.290104] acpiphp: Slot [15] registered [ 0.291080] acpiphp: Slot [16] registered [ 0.293095] acpiphp: Slot [17] registered [ 0.294078] acpiphp: Slot [18] registered [ 0.295077] acpiphp: Slot [19] registered [ 0.296053] acpiphp: Slot [20] registered [ 0.297051] acpiphp: Slot [21] registered [ 0.298055] acpiphp: Slot [22] registered [ 0.298932] acpiphp: Slot [23] registered [ 0.301074] acpiphp: Slot [24] registered [ 0.302094] acpiphp: Slot [25] registered [ 0.304075] acpiphp: Slot [26] registered [ 0.305100] acpiphp: Slot [27] registered [ 0.307128] acpiphp: Slot [28] registered [ 0.309088] acpiphp: Slot [29] registered [ 0.310146] acpiphp: Slot [30] registered [ 0.311241] acpiphp: Slot [31] registered [ 0.313051] PCI host bridge to bus 0000:00 [ 0.314013] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.316021] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.318026] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.320016] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.324045] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.327131] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.329168] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.334000] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.337131] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.345010] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.348445] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.350012] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.352015] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.353010] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.355495] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.357494] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.359034] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.362752] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.367012] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.382037] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.389024] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.396884] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.411017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.432022] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.481021] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.492258] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.502012] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.508014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.529017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.539760] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.545018] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.551013] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.568016] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.579890] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.588014] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.594016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.608019] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.616436] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.620011] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.624011] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.638017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.647747] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.653018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.660015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.673016] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.685813] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.692058] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.694321] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.697400] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.699176] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.705093] iommu: Default domain type: Passthrough [ 0.707381] SCSI subsystem initialized [ 0.709139] ACPI: bus type USB registered [ 0.710126] usbcore: registered new interface driver usbfs [ 0.712062] usbcore: registered new interface driver hub [ 0.713085] usbcore: registered new device driver usb [ 0.714461] pps_core: LinuxPPS API ver. 1 registered [ 0.716011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.718072] PTP clock support registered [ 0.721055] EDAC MC: Ver: 3.0.0 [ 0.722671] PCI: Using ACPI for IRQ routing [ 0.724157] NetLabel: Initializing [ 0.725008] NetLabel: domain hash size = 128 [ 0.727015] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.729085] NetLabel: unlabeled traffic allowed by default [ 0.731506] vgaarb: loaded [ 0.732414] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.734019] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.741000] clocksource: Switched to clocksource kvm-clock [ 0.839188] VFS: Disk quotas dquot_6.6.0 [ 0.841050] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.842604] *** VALIDATE ramfs *** [ 0.844035] *** VALIDATE hugetlbfs *** [ 0.845690] pnp: PnP ACPI init [ 0.848088] pnp: PnP ACPI: found 6 devices [ 0.862400] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.865715] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.868120] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.869993] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.871992] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.874193] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.876320] NET: Registered protocol family 2 [ 0.879133] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.884173] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.886596] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.891969] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.895266] TCP: Hash tables configured (established 65536 bind 65536) [ 0.898155] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.901440] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.904494] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.907603] NET: Registered protocol family 1 [ 0.911362] RPC: Registered named UNIX socket transport module. [ 0.914127] RPC: Registered udp transport module. [ 0.915969] RPC: Registered tcp transport module. [ 0.917661] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.920090] NET: Registered protocol family 44 [ 0.921839] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.925742] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.928081] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.930939] PCI: CLS 0 bytes, default 64 [ 0.933289] Unpacking initramfs... [ 2.382640] debug: unmapping init [mem 0xffff94587cc54000-0xffff94587ffbffff] [ 2.385765] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.387666] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.389946] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229833f6470, max_idle_ns: 440795327230 ns [ 2.970114] Initialise system trusted keyrings [ 2.973391] Key type blacklist registered [ 2.976219] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.992691] zbud: loaded [ 2.998938] *** VALIDATE nfs *** [ 3.000864] *** VALIDATE nfs4 *** [ 3.003103] pstore: using deflate compression [ 3.008405] Platform Keyring initialized [ 3.157926] NET: Registered protocol family 38 [ 3.161981] Key type asymmetric registered [ 3.163988] Asymmetric key parser 'x509' registered [ 3.166530] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.171219] io scheduler mq-deadline registered [ 3.173022] io scheduler kyber registered [ 3.174864] io scheduler bfq registered [ 3.176771] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.181102] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.184194] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.188977] ACPI: Power Button [PWRF] [ 3.196588] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.205232] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.219330] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.226969] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.248936] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.278598] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.309059] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.313853] Non-volatile memory driver v1.3 [ 3.315770] Linux agpgart interface v0.103 [ 3.356025] virtio_blk virtio1: [vda] 135128 512-byte logical blocks (69.2 MB/66.0 MiB) [ 3.358844] vda: detected capacity change from 0 to 69185536 [ 3.401809] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.405625] vdb: detected capacity change from 0 to 1073741824 [ 3.440290] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.445264] vdc: detected capacity change from 0 to 2621440000 [ 3.476136] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.479253] vdd: detected capacity change from 0 to 2621440000 [ 3.506170] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.510987] vde: detected capacity change from 0 to 4294967296 [ 3.534661] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.538360] vdf: detected capacity change from 0 to 4294967296 [ 3.551679] libphy: Fixed MDIO Bus: probed [ 3.562581] usbcore: registered new interface driver usbserial_generic [ 3.565375] usbserial: USB Serial support registered for generic [ 3.569995] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.577089] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.580040] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.583317] mousedev: PS/2 mouse device common for all mice [ 3.587274] rtc_cmos 00:05: RTC can wake from S4 [ 3.588540] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.589929] rtc_cmos 00:05: registered as rtc0 [ 3.589971] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.590016] intel_pstate: CPU model not supported [ 3.597255] hid: raw HID events driver (C) Jiri Kosina [ 3.606789] usbcore: registered new interface driver usbhid [ 3.608166] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.610429] usbhid: USB HID core driver [ 3.610667] drop_monitor: Initializing network drop monitor service [ 3.610795] Initializing XFRM netlink socket [ 3.611398] NET: Registered protocol family 10 [ 3.612672] Segment Routing with IPv6 [ 3.612797] NET: Registered protocol family 17 [ 3.615488] mpls_gso: MPLS GSO support [ 3.630329] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.640791] RAS: Correctable Errors collector initialized. [ 3.643675] AVX version of gcm_enc/dec engaged. [ 3.645636] AES CTR mode by8 optimization enabled [ 3.762904] sched_clock: Marking stable (3762595178, 0)->(4641757657, -879162479) [ 3.769906] registered taskstats version 1 [ 3.772625] Loading compiled-in X.509 certificates [ 3.775094] zswap: loaded using pool lzo/zbud [ 3.835862] Key type big_key registered [ 3.854284] Key type encrypted registered [ 3.856267] ima: No TPM chip found, activating TPM-bypass! [ 3.858739] ima: Allocated hash algorithm: sha1 [ 3.860638] ima: No architecture policies found [ 3.862692] evm: Initialising EVM extended attributes: [ 3.865766] evm: security.selinux [ 3.867751] evm: security.ima [ 3.868994] evm: security.capability [ 3.870211] evm: HMAC attrs: 0x1 [ 3.872448] rtc_cmos 00:05: setting system clock to 2026-03-10 00:17:14 UTC (1773101834) [ 3.884943] debug: unmapping init [mem 0xffffffff9e603000-0xffffffff9e7fffff] [ 3.888214] debug: unmapping init [mem 0xffffffff9d382000-0xffffffff9d658fff] [ 3.899431] Write protecting the kernel read-only data: 28672k [ 3.905952] debug: unmapping init [mem 0xffffffff9ba03000-0xffffffff9bbfffff] [ 3.913290] debug: unmapping init [mem 0xffffffff9c314000-0xffffffff9c3fffff] [ 3.962771] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.973579] systemd[1]: Detected virtualization kvm. [ 3.977456] systemd[1]: Detected architecture x86-64. [ 3.980278] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.019640] systemd[1]: No hostname configured. [ 4.024937] systemd[1]: Set hostname to . [ 4.027335] random: systemd: uninitialized urandom read (16 bytes read) [ 4.030702] systemd[1]: Initializing machine ID from random generator. [ 4.147255] random: ln: uninitialized urandom read (6 bytes read) [ 4.339117] random: systemd: uninitialized urandom read (16 bytes read) [ 4.341671] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 4.346315] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 4.350819] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... [ OK ] Reached target Sockets. [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... Starting Setup Virtual Console... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.638369] device-mapper: uevent: version 1.0.3 [ 5.641427] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 7.147276] virtio_net virtio0 ens2: renamed from eth0 Starting dracut initqueue hook... [ 7.199142] random: fast init done [ 7.572884] scsi host0: ata_piix [ 7.818870] scsi host1: ata_piix [ 7.820337] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 7.824642] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 12.585646] random: crng init done [ 12.589168] random: 7 urandom warning(s) missed due to ratelimiting [ 15.096838] dracut-initqueue[583]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 16.730837] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 18.380925] printk: systemd: 26 output lines suppressed due to ratelimiting [ 18.748246] SELinux: Disabled at runtime. [ 18.813338] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 18.825856] systemd[1]: Detected virtualization kvm. [ 18.827123] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 19.579841] systemd[1]: initrd-switch-root.service: Succeeded. [ 19.584965] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 19.597777] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 19.605422] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 19.610264] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 19.620426] systemd[1]: Starting Journal Service... Starting Journal Service... [ 19.631466] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. Mounting Huge Pages File System... [ OK ] Stopped target Switch Root. [ OK ] Listening on RPCbind Server Activation Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice User and Session Slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on Process Core Dump Socket. Mounting POSIX Message Queue File System... [ OK [ 20.301529] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS ] Started Forward Password Requests to Wall Directory Watch. Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target RPC Port Mapper. Mounting Kernel Debug File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-getty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting udev Coldplug all Devices... [ OK ] Stopped target Initrd Root File System. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 21.314462] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 22.090579] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 22.159042] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 22.561329] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 22.649581] EDAC sbridge: Ver: 1.1.2 [ 25.856450] Key type dns_resolver registered [ 26.732391] NFS: Registering the id_resolver key type [ 26.734458] Key type id_resolver registered [ 26.736457] Key type id_legacy registered [* ] A start job is running for Configur…-only root support (7s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started Login Service. [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg231-server login: [ 63.435923] libcfs: loading out-of-tree module taints kernel. [ 63.466091] Key type ._llcrypt registered [ 63.467395] Key type .llcrypt registered [ 63.506450] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_hostid [ 72.153552] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 72.837761] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 72.844948] alg: No test for adler32 (adler32-zlib) [ 73.862924] Lustre: Lustre: Build Version: 2.17.50_236_gea5aa0c [ 74.182990] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 75.815163] Key type lgssc registered [ 76.455803] Lustre: Echo OBD driver; http://www.lustre.org/ [ 83.691359] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 101.737048] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 107.451488] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 107.463927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 108.590428] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 108.611068] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 108.654175] Lustre: lustre-MDT0000: new disk, initializing [ 108.687266] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 108.702915] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 110.423770] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 116.902675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 116.941134] Lustre: 6476:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 116.958877] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 116.960826] Lustre: Skipped 1 previous similar message [ 117.006400] Lustre: lustre-MDT0001: new disk, initializing [ 117.040630] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 117.053513] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 117.057989] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 118.707622] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 121.530641] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 125.914845] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 126.023775] Lustre: lustre-OST0000: new disk, initializing [ 126.026218] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 126.050097] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 128.424277] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 130.542409] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 130.546979] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 130.571391] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 134.847167] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 134.890709] Lustre: lustre-OST0001: new disk, initializing [ 134.892722] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 134.910875] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 135.739946] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 135.747048] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 135.788558] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 137.580942] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 144.899881] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 150.206413] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 157.988863] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing check_logdir /tmp/testlogs/ [ 160.224586] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing yml_node [ 162.882204] Lustre: DEBUG MARKER: Client: 2.17.50.236 [ 164.364966] Lustre: DEBUG MARKER: MDS: 2.17.50.236 [ 165.707310] Lustre: DEBUG MARKER: OSS: 2.17.50.236 [ 166.656297] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Mar 9 20:19:57 EDT 2026 [ 176.855899] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 177.560933] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 191.972639] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 191.974617] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 191.982527] Lustre: Skipped 3 previous similar messages [ 191.990976] Lustre: Skipped 3 previous similar messages [ 197.088573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 197.094170] Lustre: Skipped 3 previous similar messages [ 202.207865] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 204.257374] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 206.407771] Lustre: server umount lustre-MDT0000 complete [ 209.042236] LustreError: 6468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773102039 with bad export cookie 16539255297915196790 [ 209.049203] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 209.053128] LustreError: 6468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 212.449271] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 212.449597] LustreError: 6483:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 212.460428] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 212.467859] LustreError: 6483:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 6 previous similar messages [ 212.477645] Lustre: Skipped 4 previous similar messages [ 214.498203] LustreError: 6484:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 214.510274] LustreError: 6484:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 215.353548] Lustre: server umount lustre-MDT0001 complete [ 217.567187] Lustre: server umount lustre-OST0000 complete [ 219.793849] Lustre: server umount lustre-OST0001 complete [ 223.627421] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_hostid [ 228.561384] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 249.687782] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 249.841753] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 249.862782] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 249.909870] Lustre: lustre-MDT0000: new disk, initializing [ 249.948310] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 249.961483] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 252.083818] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 258.431560] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 258.514934] Lustre: 14943:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 258.556760] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 258.559872] Lustre: Skipped 1 previous similar message [ 258.600170] Lustre: lustre-MDT0001: new disk, initializing [ 258.650743] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 258.665828] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 258.669666] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 260.777143] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 265.257798] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 267.253457] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 271.099465] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 273.136335] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 273.204110] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 274.826681] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 279.269512] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 281.110349] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 281.216458] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 283.275654] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 283.395255] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 288.739267] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 288.746509] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 288.755165] LustreError: Skipped 1 previous similar message [ 288.757653] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 288.765670] Lustre: Skipped 2 previous similar messages [ 288.776958] Lustre: Skipped 3 previous similar messages [ 290.421591] Lustre: server umount lustre-OST0000 complete [ 293.856454] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 293.865209] Lustre: Skipped 1 previous similar message [ 298.677296] Lustre: server umount lustre-MDT0000 complete [ 298.978902] LustreError: 14957:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 298.987722] LustreError: 14957:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 301.191714] LustreError: 15723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773102131 with bad export cookie 16539255297915198302 [ 301.194303] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 301.198109] LustreError: 15723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 301.401665] Lustre: server umount lustre-MDT0001 complete [ 304.015558] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 20:22:14 (1773102134) [ 305.829274] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 306.966301] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 20:22:17 (1773102137) [ 310.999847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 311.254212] LustreError: 18948:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 311.298649] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 311.302795] Lustre: Skipped 1 previous similar message [ 313.525461] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 316.388759] LustreError: 18949:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 318.800503] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 321.640254] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 325.147835] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 327.534789] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 331.617126] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 334.775020] hrtimer: interrupt took 3976848 ns [ 335.477346] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 339.028988] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 349.668739] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 349.669433] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 349.673403] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 349.673413] Lustre: Skipped 2 previous similar messages [ 349.679319] Lustre: Skipped 1 previous similar message [ 353.117733] Lustre: server umount lustre-OST0000 complete [ 359.904977] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 359.905462] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 361.820357] Lustre: server umount lustre-MDT0000 complete [ 364.388934] LustreError: 18928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773102195 with bad export cookie 16539255297915199191 [ 364.393818] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 364.396479] LustreError: 18928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 364.616451] Lustre: server umount lustre-MDT0001 complete [ 373.479528] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 375.321400] Key type lgssc unregistered [ 375.569648] LNet: 22126:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 375.575906] LNetError: 22126:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 375.587329] LNet: Removed LNI 192.168.202.131@tcp [ 376.152311] Key type .llcrypt unregistered [ 376.154146] Key type ._llcrypt unregistered [ 390.638876] Key type ._llcrypt registered [ 390.641036] Key type .llcrypt registered [ 390.704990] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 391.155876] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 391.173554] alg: No test for adler32 (adler32-zlib) [ 392.133374] Lustre: Lustre: Build Version: 2.17.50_236_gea5aa0c [ 392.318961] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 393.935157] Key type lgssc registered [ 394.580419] Lustre: Echo OBD driver; http://www.lustre.org/ [ 399.397069] loop: module loaded [ 399.427514] loop0: detected capacity change from 0 to 204800000 [ 399.545712] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 399.550806] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 399.727342] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 403.616523] loop0: detected capacity change from 0 to 204800000 [ 403.729391] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 403.737759] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 403.746906] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 407.468947] loop0: detected capacity change from 0 to 204800000 [ 411.460272] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 411.497575] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 412.717477] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 412.758302] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 412.817673] Lustre: lustre-MDT0000: new disk, initializing [ 412.883656] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 412.909382] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 416.792230] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 423.419668] loop1: detected capacity change from 0 to 204800000 [ 427.796270] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 428.118554] Lustre: lustre-OST07c6: new disk, initializing [ 428.124173] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 428.127624] Lustre: Skipped 1 previous similar message [ 428.204480] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 430.121268] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 430.131064] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 430.165308] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 433.198466] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 438.285522] Lustre: 24258:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 438.289160] Lustre: 24258:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 438.293291] Lustre: 24258:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 438.298035] Lustre: 24258:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 438.300977] Lustre: 24258:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 438.306145] Lustre: 24258:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 452.149171] Lustre: 24256:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 452.158135] Lustre: 24256:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 452.162581] Lustre: 24256:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 452.172756] Lustre: 24256:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/4 [ 452.176254] Lustre: 24256:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 452.180745] Lustre: 24256:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 456.461606] Lustre: 25709:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 456.468235] Lustre: 25709:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 1 previous similar message [ 456.472325] Lustre: 25709:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 456.476629] Lustre: 25709:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 456.481494] Lustre: 25709:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 456.487215] Lustre: 25709:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 456.492261] Lustre: 25709:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/0, punch: 0/0/0, quota 4/150/0 [ 456.496482] Lustre: 25709:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 456.503437] Lustre: 25709:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 456.508488] Lustre: 25709:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 456.513563] Lustre: 25709:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 456.515918] Lustre: 25709:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 459.633602] Lustre: 24258:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 459.641783] Lustre: 24258:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 1 previous similar message [ 459.650964] Lustre: 24258:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 459.655687] Lustre: 24258:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 459.664823] Lustre: 24258:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 459.673368] Lustre: 24258:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 459.677463] Lustre: 24258:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/1 [ 459.684380] Lustre: 24258:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 459.687674] Lustre: 24258:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 459.693063] Lustre: 24258:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 459.699145] Lustre: 24258:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 459.705282] Lustre: 24258:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 463.964172] Lustre: 24257:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 463.981230] Lustre: 24257:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 1 previous similar message [ 463.985753] Lustre: 24257:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 463.989252] Lustre: 24257:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 463.993266] Lustre: 24257:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 464.002063] Lustre: 24257:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 464.007447] Lustre: 24257:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/2 [ 464.011827] Lustre: 24257:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 464.017797] Lustre: 24257:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 464.029990] Lustre: 24257:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 464.038448] Lustre: 24257:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 464.041585] Lustre: 24257:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 474.548181] Lustre: 25709:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 474.568069] Lustre: 25709:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 5 previous similar messages [ 474.579865] Lustre: 25709:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 474.587445] Lustre: 25709:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 474.599124] Lustre: 25709:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 474.605971] Lustre: 25709:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 474.610212] Lustre: 25709:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/0, punch: 0/0/0, quota 4/150/0 [ 474.615707] Lustre: 25709:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 474.622457] Lustre: 25709:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 474.629117] Lustre: 25709:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 474.635142] Lustre: 25709:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 474.639830] Lustre: 25709:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 520.899161] Lustre: server umount lustre-OST07c6 complete [ 536.666179] Lustre: server umount lustre-MDT0000 complete [ 547.716150] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 550.194758] Key type lgssc unregistered [ 550.466568] LNet: 27773:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 550.472119] LNetError: 27773:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 550.488361] LNet: Removed LNI 192.168.202.131@tcp [ 551.200747] Key type .llcrypt unregistered [ 551.203214] Key type ._llcrypt unregistered [ 562.577912] Key type ._llcrypt registered [ 562.579421] Key type .llcrypt registered [ 562.659715] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 563.230316] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 563.251976] alg: No test for adler32 (adler32-zlib) [ 564.237348] Lustre: Lustre: Build Version: 2.17.50_236_gea5aa0c [ 564.465244] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 566.095286] Key type lgssc registered [ 566.958739] Lustre: Echo OBD driver; http://www.lustre.org/ [ 573.602331] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 573.619715] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 575.144894] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 578.570353] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 584.673051] LustreError: 29007:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 584.679935] LustreError: 29007:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 585.012491] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 585.407458] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 589.267697] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 593.603256] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 596.075337] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 601.068029] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 601.301497] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 605.497747] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 609.595092] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 619.753077] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 635.367170] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 635.368057] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 635.369320] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 635.375347] Lustre: Skipped 1 previous similar message [ 635.386565] LustreError: Skipped 1 previous similar message [ 636.570530] Lustre: server umount lustre-OST0000 complete [ 640.480739] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 640.483413] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 640.488386] Lustre: Skipped 1 previous similar message [ 640.491490] Lustre: Skipped 1 previous similar message [ 645.600486] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 645.604221] Lustre: Skipped 2 previous similar messages [ 650.720985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 650.724685] Lustre: Skipped 1 previous similar message [ 653.279210] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 653.415437] Lustre: server umount lustre-MDT0000 complete [ 655.821301] LustreError: 28986:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773102486 with bad export cookie 8057235567765656197 [ 655.826698] LustreError: 28986:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 655.828311] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 656.054409] Lustre: server umount lustre-MDT0001 complete [ 661.775226] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 663.641845] Key type lgssc unregistered [ 663.872557] LNet: 32414:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 663.878726] LNetError: 32414:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 664.936076] LNet: Removed LNI 192.168.202.131@tcp [ 665.454295] Key type .llcrypt unregistered [ 665.456624] Key type ._llcrypt unregistered [ 675.606917] Key type ._llcrypt registered [ 675.608151] Key type .llcrypt registered [ 675.669437] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 676.231280] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 676.241148] alg: No test for adler32 (adler32-zlib) [ 677.278638] Lustre: Lustre: Build Version: 2.17.50_236_gea5aa0c [ 677.431512] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 679.080909] Key type lgssc registered [ 679.991411] Lustre: Echo OBD driver; http://www.lustre.org/ [ 684.952494] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 684.970245] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 686.324528] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 688.480306] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 692.889977] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 693.120607] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 695.089089] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 697.633683] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 699.397743] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 702.951468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 703.233278] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 707.084994] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 710.278133] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 710.452689] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 719.841189] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 719.846640] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 719.854555] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 720.866194] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 720.868724] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 720.870237] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 720.881616] Lustre: Skipped 1 previous similar message [ 722.257233] Lustre: server umount lustre-OST0000 complete [ 725.988763] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 725.991477] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 725.994179] Lustre: Skipped 1 previous similar message [ 731.106192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 731.110430] Lustre: Skipped 1 previous similar message [ 736.224452] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 736.229015] Lustre: Skipped 1 previous similar message [ 739.295156] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 739.432344] Lustre: server umount lustre-MDT0000 complete [ 741.346968] LustreError: 33645:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 741.358458] LustreError: 33645:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 741.620381] LustreError: 33626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773102572 with bad export cookie 12884940265717337854 [ 741.625176] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 741.629481] LustreError: 33626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 741.817431] Lustre: server umount lustre-MDT0001 complete [ 747.388937] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 749.334870] Key type lgssc unregistered [ 749.579172] LNet: 36826:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 749.586812] LNetError: 36826:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 749.596687] LNet: Removed LNI 192.168.202.131@tcp [ 750.191463] Key type .llcrypt unregistered [ 750.194827] Key type ._llcrypt unregistered [ 765.339865] Key type ._llcrypt registered [ 765.341414] Key type .llcrypt registered [ 765.429324] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 766.039057] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 766.061795] alg: No test for adler32 (adler32-zlib) [ 767.025176] Lustre: Lustre: Build Version: 2.17.50_236_gea5aa0c [ 767.182699] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 768.791317] Key type lgssc registered [ 769.577068] Lustre: Echo OBD driver; http://www.lustre.org/ [ 774.583857] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 774.602824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 775.895628] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 778.175253] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 783.332613] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 783.571374] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 786.566472] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 789.928426] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 792.183531] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 796.351964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 796.542369] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 800.170273] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 800.614963] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 803.160702] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 814.052648] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 814.059491] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 814.063521] Lustre: Skipped 2 previous similar messages [ 816.003110] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.31@tcp (stopping) [ 818.278098] Lustre: server umount lustre-MDT0000 complete [ 819.173168] LustreError: 38198:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 819.186415] LustreError: 38198:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 4 previous similar messages [ 820.503780] LustreError: 38185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773102651 with bad export cookie 11694847641569819401 [ 820.506326] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 820.511679] LustreError: 38185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 821.106839] LustreError: 38200:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 821.115321] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.31@tcp (stopping) [ 824.288355] LustreError: 38198:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 824.288553] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 824.302554] Lustre: Skipped 2 previous similar messages [ 824.305480] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 826.712360] Lustre: server umount lustre-MDT0001 complete [ 845.910990] Lustre: server umount lustre-OST0000 complete [ 853.733218] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 855.054730] Key type lgssc unregistered [ 855.221908] LNet: 41712:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 855.231530] LNetError: 41712:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 855.248743] LNet: Removed LNI 192.168.202.131@tcp [ 855.669535] Key type .llcrypt unregistered [ 855.671542] Key type ._llcrypt unregistered [ 867.073078] Key type ._llcrypt registered [ 867.074515] Key type .llcrypt registered [ 867.123045] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 867.510940] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 867.551282] alg: No test for adler32 (adler32-zlib) [ 868.441561] Lustre: Lustre: Build Version: 2.17.50_236_gea5aa0c [ 868.536613] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 870.127162] Key type lgssc registered [ 870.671711] Lustre: Echo OBD driver; http://www.lustre.org/ [ 874.219699] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 874.230664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 875.467729] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 877.215623] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 881.301878] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 881.496483] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 883.445649] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 885.989586] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 887.636912] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 890.891955] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 891.033604] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 893.652471] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 896.039633] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 901.285399] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 906.618170] Lustre: lustre-OST0000: Not available for connect from 192.168.202.31@tcp (stopping) [ 906.723591] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 906.724027] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 906.732715] Lustre: Skipped 1 previous similar message [ 909.982305] Lustre: server umount lustre-OST0000 complete [ 926.624704] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 926.625709] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 926.628973] Lustre: Skipped 1 previous similar message [ 926.631577] Lustre: Skipped 2 previous similar messages [ 929.769816] Lustre: server umount lustre-MDT0000 complete [ 931.431739] LustreError: 43071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773102762 with bad export cookie 12797027713224945515 [ 931.437197] LustreError: 43071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 931.437721] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 937.759699] Lustre: server umount lustre-MDT0001 complete [ 943.071799] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 944.491332] Key type lgssc unregistered [ 944.662719] LNet: 46444:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 944.667279] LNetError: 46444:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 944.677043] LNet: Removed LNI 192.168.202.131@tcp [ 945.117502] Key type .llcrypt unregistered [ 945.119161] Key type ._llcrypt unregistered [ 958.381521] Key type ._llcrypt registered [ 958.382779] Key type .llcrypt registered [ 958.454721] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 958.972744] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 959.074361] alg: No test for adler32 (adler32-zlib) [ 959.978835] Lustre: Lustre: Build Version: 2.17.50_236_gea5aa0c [ 960.088861] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 961.703145] Key type lgssc registered [ 962.281402] Lustre: Echo OBD driver; http://www.lustre.org/ [ 965.857823] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 965.866957] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 967.141214] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 968.838597] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 973.090328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 973.288395] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 975.249675] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 978.098828] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 979.911355] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 983.587122] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 983.788852] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 986.672083] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 989.523419] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 993.000425] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1002.524707] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1004.382708] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1015.480282] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-03-0920h34m05s [ 1016.245721] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1018.852501] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1018.857021] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1018.857194] Lustre: Skipped 2 previous similar messages [ 1018.860955] Lustre: Skipped 1 previous similar message [ 1019.872539] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1019.876729] Lustre: Skipped 2 previous similar messages [ 1021.816431] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.31@tcp (stopping) [ 1023.379543] Lustre: server umount lustre-MDT0000 complete [ 1025.073754] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1026.020094] LustreError: 48531:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1026.027663] LustreError: 48531:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 4 previous similar messages [ 1027.888127] LustreError: 47818:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1028.638662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1028.702353] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1028.833300] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1028.871443] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1029.864102] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1030.685539] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1032.144350] Lustre: DEBUG MARKER: Wait for df (36012) ... [ 1032.819281] Lustre: DEBUG MARKER: done [ 1034.170094] Lustre: server umount lustre-OST0000 complete [ 1040.353737] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1040.356492] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1040.356614] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1040.361946] Lustre: Skipped 1 previous similar message [ 1040.363831] Lustre: Skipped 1 previous similar message [ 1045.508806] Lustre: server umount lustre-MDT0000 complete [ 1047.319899] LustreError: 47803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773102877 with bad export cookie 16247222301833145889 [ 1047.322777] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1047.323783] LustreError: 47803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1056.814341] Lustre: server umount lustre-MDT0001 complete [ 1062.470918] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 1064.114738] Key type lgssc unregistered [ 1064.288459] LNet: 52389:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1064.292386] LNetError: 52389:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1064.300476] LNet: Removed LNI 192.168.202.131@tcp [ 1064.722194] Key type .llcrypt unregistered [ 1064.723535] Key type ._llcrypt unregistered [ 1081.708309] Key type ._llcrypt registered [ 1081.709459] Key type .llcrypt registered [ 1081.751840] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 1082.075454] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1082.084351] alg: No test for adler32 (adler32-zlib) [ 1082.949349] Lustre: Lustre: Build Version: 2.17.50_236_gea5aa0c [ 1083.051869] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 1084.639146] Key type lgssc registered [ 1085.072415] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1088.303790] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1088.314814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1089.456368] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1089.468455] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1089.536115] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1091.094910] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1094.838465] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1094.886484] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1094.899371] Lustre: 54554:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1095.001579] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1096.590593] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1098.781837] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1100.124981] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1102.776325] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1102.855960] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1102.941334] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1105.169426] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1107.288372] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1109.017761] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1117.448603] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-03-0920h35m48s [ 1118.115410] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1120.999850] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1121.998821] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1122.040356] LustreError: 56747:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1142.144383] Lustre: lustre-MDT0000: Client be7532f9-d38b-44ad-9b68-1bb0afb4ea15 (at 192.168.202.31@tcp) reconnecting [ 1162.090284] LustreError: 56747:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1163.146868] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1164.253270] Lustre: DEBUG MARKER: done [ 1167.846664] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 1167.852146] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1167.859599] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1170.405454] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1170.408390] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1170.412661] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1170.418862] Lustre: Skipped 1 previous similar message [ 1172.438165] Lustre: server umount lustre-OST0000 complete [ 1175.520651] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1175.521371] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1175.525774] Lustre: Skipped 1 previous similar message [ 1180.378882] Lustre: server umount lustre-MDT0000 complete [ 1182.197141] LustreError: 56978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773103012 with bad export cookie 18057489017452360540 [ 1182.200186] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1182.204807] LustreError: 56978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1182.365687] Lustre: server umount lustre-MDT0001 complete [ 1187.168830] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 1188.635809] Key type lgssc unregistered [ 1188.799398] LNet: 58525:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1188.806281] LNetError: 58525:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1188.819500] LNet: Removed LNI 192.168.202.131@tcp [ 1189.235734] Key type .llcrypt unregistered [ 1189.238605] Key type ._llcrypt unregistered [ 1212.121485] Key type ._llcrypt registered [ 1212.122855] Key type .llcrypt registered [ 1212.199461] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 1212.771546] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1212.789127] alg: No test for adler32 (adler32-zlib) [ 1213.684698] Lustre: Lustre: Build Version: 2.17.50_236_gea5aa0c [ 1213.794990] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 1215.391228] Key type lgssc registered [ 1215.928958] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1217.521086] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1217.533859] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1218.669765] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1218.684866] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1218.770517] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1219.216541] Lustre: Failing over lustre-MDT0000 [ 1219.373811] Lustre: server umount lustre-MDT0000 complete [ 1222.595094] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 20:37:33 (1773103053) [ 1225.860769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1226.099947] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1228.027193] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1231.985274] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1232.052858] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1232.075136] Lustre: 61282:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1232.204382] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1233.997787] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1236.766460] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1238.444574] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1241.842362] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1241.948247] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1242.037538] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1244.672847] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1247.174111] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1248.281105] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1255.907923] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1258.467139] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1258.469476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1258.473237] Lustre: Skipped 1 previous similar message [ 1263.584795] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1263.587892] Lustre: Skipped 4 previous similar messages [ 1264.374659] Lustre: server umount lustre-MDT0000 complete [ 1266.220124] LustreError: 61999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773103096 with bad export cookie 15698701163481712915 [ 1266.223422] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1266.227118] LustreError: 61999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1266.371667] Lustre: server umount lustre-MDT0001 complete [ 1268.042344] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1270.352158] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1274.176343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1275.872305] LustreError: 60206:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000005c8036b1 x1859233297499008/t0(0) o250->MGC192.168.202.131@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1276.021728] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1276.026097] Lustre: Skipped 1 previous similar message [ 1276.064475] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1276.109093] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1277.774766] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1281.391929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1281.506627] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1281.509284] Lustre: Skipped 1 previous similar message [ 1283.102299] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1285.215122] Lustre: 60207:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773103099/real 1773103099] req@000000007d41e598 x1859233297498624/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1773103115 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1285.401489] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1286.629418] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1286.892219] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1287.647116] Lustre: 60208:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773103102/real 1773103102] req@0000000013dde549 x1859233297498752/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1773103118 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1288.678199] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1290.335276] Lustre: 60209:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773103104/real 1773103104] req@00000000413708ad x1859233297498880/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1773103120 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1291.744396] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1291.749476] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1291.757574] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1292.771525] Lustre: 60209:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773103106/real 1773103106] req@00000000bb75b4e6 x1859233297499136/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1773103122 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1293.797649] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1293.803843] Lustre: Skipped 1 previous similar message [ 1295.471302] Lustre: server umount lustre-MDT0000 complete [ 1297.111323] LustreError: 62909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773103127 with bad export cookie 15698701163481715183 [ 1297.119285] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1297.264679] Lustre: server umount lustre-MDT0001 complete [ 1300.321184] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1304.590465] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1306.719155] LustreError: 67150:0:(import.c:337:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 1306.722687] LustreError: 67150:0:(import.c:361:ptlrpc_invalidate_import()) @@@ still on sending list req@0000000015af36d5 x1859233297518080/t0(0) o250->MGC192.168.202.131@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1773103137 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1306.730247] LustreError: 67150:0:(import.c:371:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 1307.039844] LustreError: 60206:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000e1e6d595 x1859233297518720/t0(0) o250->MGC192.168.202.131@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1307.153245] LustreError: 62902:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1307.161114] LustreError: 62902:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 1307.188955] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1307.191348] Lustre: Skipped 1 previous similar message [ 1307.211785] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1308.732656] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1312.226894] LustreError: 65931:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1312.227555] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1312.252588] LustreError: 65931:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 1312.677070] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1313.894212] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1314.611247] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1314.911236] Lustre: 60209:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773103129/real 1773103129] req@00000000583947dc x1859233297518336/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1773103145 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1316.885750] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1318.414200] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1322.976129] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1322.980098] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1322.985925] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1322.988560] Lustre: Skipped 3 previous similar messages [ 1326.557642] Lustre: server umount lustre-MDT0000 complete [ 1328.189903] LustreError: 61273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773103158 with bad export cookie 15698701163481716562 [ 1328.192754] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1328.201731] LustreError: 61273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 1328.353973] Lustre: server umount lustre-MDT0001 complete [ 1331.193914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1332.806238] Lustre: DEBUG MARKER: files compared the same [ 1343.912720] Lustre: server umount lustre-OST0000 complete [ 1349.990879] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 1351.141328] Key type lgssc unregistered [ 1351.279752] LNet: 70195:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1351.283474] LNetError: 70195:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1351.293436] LNet: Removed LNI 192.168.202.131@tcp [ 1351.606815] Key type .llcrypt unregistered [ 1351.607863] Key type ._llcrypt unregistered [ 1361.041152] Key type ._llcrypt registered [ 1361.042580] Key type .llcrypt registered [ 1361.081137] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 1361.547906] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1361.567541] alg: No test for adler32 (adler32-zlib) [ 1362.452484] Lustre: Lustre: Build Version: 2.17.50_236_gea5aa0c [ 1362.549391] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 1364.143141] Key type lgssc registered [ 1364.611903] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1367.645435] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1367.656119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1368.862737] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1370.240476] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug +malloc all [ 1373.385047] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1373.504653] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1374.992871] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug +malloc all [ 1377.193162] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1378.695280] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1381.240802] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1381.354097] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1383.482692] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug +malloc all [ 1385.409312] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1393.636731] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1403.872546] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1403.872734] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1403.880762] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1403.883672] Lustre: Skipped 1 previous similar message [ 1408.991909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1408.994537] Lustre: Skipped 1 previous similar message [ 1413.087249] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1413.142270] Lustre: server umount lustre-OST0000 complete [ 1419.232226] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1419.232514] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1419.237291] Lustre: Skipped 2 previous similar messages [ 1420.824914] Lustre: server umount lustre-MDT0000 complete [ 1422.218328] LustreError: 73185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773103252 with bad export cookie 18404649115501799702 [ 1422.218587] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1422.223266] LustreError: 73185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1422.355340] Lustre: server umount lustre-MDT0001 complete [ 1426.058505] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 1427.132668] Key type lgssc unregistered [ 1427.263515] LNet: 74773:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1427.266781] LNetError: 74773:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1427.275422] LNet: Removed LNI 192.168.202.131@tcp [ 1427.586434] Key type .llcrypt unregistered [ 1427.587745] Key type ._llcrypt unregistered [ 1437.190853] Key type ._llcrypt registered [ 1437.192012] Key type .llcrypt registered [ 1437.232643] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 1437.591788] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1437.652494] alg: No test for adler32 (adler32-zlib) [ 1438.517642] Lustre: Lustre: Build Version: 2.17.50_236_gea5aa0c [ 1438.595425] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 1440.175168] Key type lgssc registered [ 1440.600484] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1443.566227] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1443.573683] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1459.039257] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1464.159270] LustreError: 76124:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.131@tcp: failed processing log, type 4: rc = -110 [ 1490.783232] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1492.678504] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1494.501603] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1497.267311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1497.319071] Lustre: *** cfs_fail_loc=706, val=0*** [ 1497.477820] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1497.504249] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 1498.910951] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1502.051862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1502.214365] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1503.654759] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1505.533063] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1506.877555] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1507.807950] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1507.812313] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1507.817811] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1509.711150] Lustre: server umount lustre-OST0000 complete [ 1511.392522] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1511.393150] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1511.398154] Lustre: Skipped 1 previous similar message [ 1516.512265] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1516.514802] Lustre: Skipped 2 previous similar messages [ 1517.148418] Lustre: server umount lustre-MDT0000 complete [ 1518.483413] LustreError: 78446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773103349 with bad export cookie 3622803852861392278 [ 1518.486675] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1518.487880] LustreError: 78446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1528.862455] Lustre: server umount lustre-MDT0001 complete [ 1532.543883] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 1533.652064] Key type lgssc unregistered [ 1533.797515] LNet: 79383:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1533.800633] LNetError: 79383:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1533.806383] LNet: Removed LNI 192.168.202.131@tcp [ 1534.108771] Key type .llcrypt unregistered [ 1534.110346] Key type ._llcrypt unregistered [ 1543.387478] Key type ._llcrypt registered [ 1543.388518] Key type .llcrypt registered [ 1543.432076] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 1543.827231] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1543.854511] alg: No test for adler32 (adler32-zlib) [ 1544.717454] Lustre: Lustre: Build Version: 2.17.50_236_gea5aa0c [ 1544.802883] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 1546.391190] Key type lgssc registered [ 1546.787187] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1549.676119] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1549.687612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1552.203366] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1555.241120] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1555.432758] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1556.797410] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1559.783784] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1559.955743] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1562.021235] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1565.152166] LustreError: 81432:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1565.158522] LustreError: 81432:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 7 previous similar messages [ 1565.190717] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1565.197060] Lustre: MGS: Client 773744e7-c07a-45df-94a3-c9f347a04e8b (at 0@lo) reconnecting [ 1565.287650] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1565.325065] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 1566.681968] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1575.393103] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1575.395297] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1575.399920] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1576.929222] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1576.929947] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1576.930916] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1576.940178] Lustre: Skipped 1 previous similar message [ 1580.629334] Lustre: server umount lustre-OST0000 complete [ 1582.048973] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1582.049311] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1582.049841] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1582.052745] Lustre: Skipped 1 previous similar message [ 1587.168098] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1587.170479] Lustre: Skipped 2 previous similar messages [ 1592.288131] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1592.290976] Lustre: Skipped 1 previous similar message [ 1596.383095] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1596.421787] Lustre: server umount lustre-MDT0000 complete [ 1597.826404] Lustre: server umount lustre-MDT0001 complete [ 1599.093923] Lustre: server umount MGS complete [ 1603.168136] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 20:43:54 (1773103434) [ 1614.039660] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_hostid [ 1617.671737] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 1630.522104] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1631.949423] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1636.295138] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1636.420773] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 1636.444688] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1636.488169] Lustre: lustre-MDT0001: new disk, initializing [ 1636.532371] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1636.553361] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1636.560520] Lustre: 88511:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 1636.564298] LustreError: 88511:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 1638.101460] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1642.661584] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1642.766971] Lustre: lustre-OST0000: new disk, initializing [ 1642.769487] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1642.796572] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1644.923342] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1645.728176] Lustre: 89551:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@0000000081939f9c x1859233644578304/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1773103492 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1648.307772] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1650.861030] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1650.866806] Lustre: MGS: Client c0dc73e4-f08d-405e-9e21-9c2d0a578676 (at 0@lo) reconnecting [ 1650.884244] Lustre: 90193:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1650.893114] LustreError: 88496:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 1650.897092] LustreError: 88496:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 1650.902663] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1650.946325] Lustre: lustre-MDT0000: new disk, initializing [ 1651.021908] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1651.034348] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1652.064769] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1652.068688] Lustre: 90223:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@000000006d7e6834 x1859233644584576/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1773103498 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1652.559872] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1654.114979] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1654.118368] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1654.186855] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 1671.648325] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1671.651614] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1671.657048] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1671.659964] Lustre: Skipped 2 previous similar messages [ 1677.415145] Lustre: server umount lustre-OST0000 complete [ 1683.937972] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1683.938237] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1683.945278] Lustre: Skipped 2 previous similar messages [ 1689.056136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1689.059095] Lustre: Skipped 5 previous similar messages [ 1693.151151] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1693.198824] Lustre: server umount lustre-MDT0000 complete [ 1694.177643] LustreError: 88485:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1694.185563] LustreError: 88485:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 1700.729944] Lustre: server umount lustre-MDT0001 complete [ 1702.176361] Lustre: server umount MGS complete [ 1706.273552] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 20:45:37 (1773103537) [ 1712.710631] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 1713.894962] Key type lgssc unregistered [ 1714.033382] LNet: 92683:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1714.037140] LNetError: 92683:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1714.051496] LNet: Removed LNI 192.168.202.131@tcp [ 1714.390098] Key type .llcrypt unregistered [ 1714.391486] Key type ._llcrypt unregistered [ 1721.928866] Key type ._llcrypt registered [ 1721.930366] Key type .llcrypt registered [ 1721.987732] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 1722.392449] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1722.430689] alg: No test for adler32 (adler32-zlib) [ 1723.298281] Lustre: Lustre: Build Version: 2.17.50_236_gea5aa0c [ 1723.399689] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 1724.991188] Key type lgssc registered [ 1725.445615] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1728.821362] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1728.830265] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1730.027970] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1731.545801] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1735.156957] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1735.329265] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1736.976272] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1739.401806] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1740.995765] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1743.728893] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1743.931371] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1746.423644] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1748.455435] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1755.172449] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 1765.345728] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1765.346349] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1765.347238] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1765.350613] Lustre: Skipped 1 previous similar message [ 1767.828868] Lustre: server umount lustre-OST0000 complete [ 1770.465728] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1770.467818] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1770.471573] Lustre: Skipped 1 previous similar message [ 1770.474284] Lustre: Skipped 1 previous similar message [ 1775.514706] Lustre: server umount lustre-MDT0000 complete [ 1775.584104] LustreError: 93863:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1775.588769] LustreError: 93863:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 1776.875889] LustreError: 93845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773103607 with bad export cookie 596299911008178711 [ 1776.879423] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1776.880798] LustreError: 93845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1777.010978] Lustre: server umount lustre-MDT0001 complete [ 1781.146035] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 1782.208361] Key type lgssc unregistered [ 1782.346501] LNet: 97060:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1782.349590] LNetError: 97060:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1782.363450] LNet: Removed LNI 192.168.202.131@tcp [ 1782.647402] Key type .llcrypt unregistered [ 1782.648631] Key type ._llcrypt unregistered [ 1789.671493] Key type ._llcrypt registered [ 1789.673211] Key type .llcrypt registered [ 1789.717700] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 1790.064238] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1790.106058] alg: No test for adler32 (adler32-zlib) [ 1790.980355] Lustre: Lustre: Build Version: 2.17.50_236_gea5aa0c [ 1791.083687] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 1792.671210] Key type lgssc registered [ 1793.108129] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1795.569332] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1795.572410] LustreError: 98105:0:(tgt_mount.c:2522:server_fill_super()) cfs_race id 716 sleeping [ 1795.580738] LustreError: 98109:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 waking [ 1795.583582] LustreError: 98105:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 awake: rc=4992 [ 1795.588984] LustreError: 98105:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 1795.593877] LustreError: 98105:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1795.599748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1796.775504] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1799.702993] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1799.825846] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1801.409698] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1803.397773] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing lsmod [ 1804.627810] LustreError: 99273:0:(tgt_mount.c:2522:server_fill_super()) cfs_race id 716 sleeping [ 1804.646138] LustreError: 99277:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 waking [ 1804.649459] LustreError: 99273:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 awake: rc=4986 [ 1804.652657] LustreError: 99273:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 1804.656690] LustreError: 99273:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1804.660348] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1804.858466] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1809.888799] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1809.890216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1809.893946] Lustre: Skipped 2 previous similar messages [ 1809.895952] Lustre: Skipped 1 previous similar message [ 1810.913214] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1810.915766] Lustre: Skipped 2 previous similar messages [ 1812.198174] Lustre: server umount lustre-MDT0000 complete [ 1813.699365] LustreError: 98124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773103644 with bad export cookie 12177904773140539457 [ 1813.703023] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1813.704408] LustreError: 98124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1815.007360] LustreError: 98575:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000004cc32108 x1859233902510080/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1815.064649] Lustre: server umount lustre-MDT0001 complete [ 1826.841422] Lustre: server umount lustre-OST0000 complete [ 1829.999757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1830.174700] LustreError: 100345:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1830.183770] LustreError: 100345:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 1830.199933] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1831.597554] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1834.644181] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1834.792814] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1836.320788] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1838.426652] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1839.921807] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1842.495269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1844.670653] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1846.639397] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1854.950909] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 1860.065197] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1860.066114] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1860.070425] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1860.072087] LustreError: Skipped 1 previous similar message [ 1860.073966] Lustre: Skipped 1 previous similar message [ 1864.913783] Lustre: server umount lustre-OST0000 complete [ 1870.305093] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1870.306248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1870.309130] Lustre: Skipped 2 previous similar messages [ 1872.407449] Lustre: server umount lustre-MDT0000 complete [ 1873.727569] LustreError: 100326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773103704 with bad export cookie 12177904773140540143 [ 1873.730188] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1873.732615] LustreError: 100326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1873.853723] Lustre: server umount lustre-MDT0001 complete [ 1877.631033] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 1878.656398] Key type lgssc unregistered [ 1878.784346] LNet: 103544:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1878.786772] LNetError: 103544:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1878.798275] LNet: Removed LNI 192.168.202.131@tcp [ 1879.115877] Key type .llcrypt unregistered [ 1879.117394] Key type ._llcrypt unregistered [ 1888.538695] Key type ._llcrypt registered [ 1888.540657] Key type .llcrypt registered [ 1888.584546] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 1888.901406] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1888.935080] alg: No test for adler32 (adler32-zlib) [ 1889.791844] Lustre: Lustre: Build Version: 2.17.50_236_gea5aa0c [ 1889.879231] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 1891.455160] Key type lgssc registered [ 1891.850703] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1894.746204] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1894.752980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1895.926541] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1897.254164] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1900.355241] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1900.495543] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1901.839828] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1903.767975] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1905.093837] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1907.660622] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1907.780327] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1909.843948] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1911.857672] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1921.126579] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 1924.426979] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 1925.345489] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 1926.625921] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1926.626565] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1926.628083] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1926.628585] LustreError: Skipped 1 previous similar message [ 1926.633913] Lustre: Skipped 1 previous similar message [ 1931.744178] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1931.746816] Lustre: Skipped 2 previous similar messages [ 1932.558675] Lustre: server umount lustre-OST0000 complete [ 1936.351863] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1936.355084] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1936.360898] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1940.183492] Lustre: server umount lustre-MDT0000 complete [ 1941.507179] LustreError: 106533:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773103772 with bad export cookie 13044954923869235372 [ 1941.510496] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1941.511553] LustreError: 106533:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1941.635947] Lustre: server umount lustre-MDT0001 complete [ 1945.466448] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 1946.559419] Key type lgssc unregistered [ 1946.695529] LNet: 108220:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1946.698272] LNetError: 108220:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1946.704294] LNet: Removed LNI 192.168.202.131@tcp [ 1947.007341] Key type .llcrypt unregistered [ 1947.008739] Key type ._llcrypt unregistered [ 1954.311196] Key type ._llcrypt registered [ 1954.312606] Key type .llcrypt registered [ 1954.363098] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 1954.737515] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1954.801451] alg: No test for adler32 (adler32-zlib) [ 1955.668412] Lustre: Lustre: Build Version: 2.17.50_236_gea5aa0c [ 1955.757615] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 1957.335161] Key type lgssc registered [ 1957.751575] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1960.684773] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1960.692495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1961.857140] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1963.291335] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1966.642934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1966.802291] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1968.294675] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1970.401199] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1971.703961] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1974.343538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1974.484745] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1976.628813] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1978.851821] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1986.789812] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 1997.280116] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1997.283532] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1997.285952] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1997.290655] Lustre: Skipped 1 previous similar message [ 1998.430372] Lustre: server umount lustre-OST0000 complete [ 2002.400948] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2002.401616] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2002.405384] Lustre: Skipped 1 previous similar message [ 2002.407931] Lustre: Skipped 1 previous similar message [ 2007.520097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2007.522899] Lustre: Skipped 2 previous similar messages [ 2012.640100] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2012.642827] Lustre: Skipped 1 previous similar message [ 2014.175125] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2014.237356] Lustre: server umount lustre-MDT0000 complete [ 2015.627795] LustreError: 109381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773103846 with bad export cookie 12302893793001291097 [ 2015.631979] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2015.635054] LustreError: 109381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2015.767115] Lustre: server umount lustre-MDT0001 complete [ 2019.750930] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 2020.884831] Key type lgssc unregistered [ 2021.020535] LNet: 112602:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2021.022644] LNetError: 112602:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2021.031297] LNet: Removed LNI 192.168.202.131@tcp [ 2021.347786] Key type .llcrypt unregistered [ 2021.348757] Key type ._llcrypt unregistered [ 2030.996219] Key type ._llcrypt registered [ 2030.997716] Key type .llcrypt registered [ 2031.045000] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 2031.481394] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2031.492637] alg: No test for adler32 (adler32-zlib) [ 2032.395228] Lustre: Lustre: Build Version: 2.17.50_236_gea5aa0c [ 2032.493662] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 2034.087186] Key type lgssc registered [ 2034.530793] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2037.559230] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2037.567833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2038.743888] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2040.168592] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2043.429982] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2043.572464] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2045.067214] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2047.044751] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2048.392248] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2050.838822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2050.955200] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2053.129953] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2055.137300] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2064.294420] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 2068.427498] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 2069.818237] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 2069.820868] Lustre: Skipped 2 previous similar messages [ 2071.403865] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 2071.407089] Lustre: Skipped 2 previous similar messages [ 2075.134029] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 2075.143912] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2079.979562] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.31@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 2079.984045] Lustre: Skipped 2 previous similar messages [ 2082.934962] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.31@tcp 192.168.0.[2,10]@tcp [ 2087.420383] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.31/24@tcp in log lustre-MDT0000 [ 2087.422991] Lustre: Skipped 2 previous similar messages [ 2090.937670] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.31/24@tcp [ 2090.939722] Lustre: Skipped 1 previous similar message [ 2100.193913] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2100.195589] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2100.201592] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2100.202937] LustreError: Skipped 2 previous similar messages [ 2100.204805] Lustre: Skipped 1 previous similar message [ 2101.262075] Lustre: server umount lustre-OST0000 complete [ 2105.313114] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2105.313386] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2105.313750] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2105.317891] Lustre: Skipped 2 previous similar messages [ 2108.765575] Lustre: server umount lustre-MDT0000 complete [ 2110.129333] LustreError: 113961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773103940 with bad export cookie 8293782178059379028 [ 2110.131804] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2110.133157] LustreError: 113961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2110.256257] Lustre: server umount lustre-MDT0001 complete [ 2113.934562] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 2114.998932] Key type lgssc unregistered [ 2115.134322] LNet: 118580:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2115.136956] LNetError: 118580:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2115.146088] LNet: Removed LNI 192.168.202.131@tcp [ 2115.453427] Key type .llcrypt unregistered [ 2115.454805] Key type ._llcrypt unregistered [ 2126.170600] Key type ._llcrypt registered [ 2126.171759] Key type .llcrypt registered [ 2126.209797] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 2126.630195] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2126.642613] alg: No test for adler32 (adler32-zlib) [ 2127.505660] Lustre: Lustre: Build Version: 2.17.50_236_gea5aa0c [ 2127.585852] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 2129.167141] Key type lgssc registered [ 2129.540455] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2132.211287] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2132.218380] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2133.383038] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 2133.388105] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2133.393303] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.31/24@tcp [ 2134.778793] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2137.907463] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2138.007941] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2138.016244] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.31/24@tcp [ 2138.031072] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2139.397433] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2141.341474] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2142.627070] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2144.969349] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2145.095424] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2147.069756] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2149.021231] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2158.439812] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 2163.681766] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2163.681843] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2163.684084] LustreError: Skipped 1 previous similar message [ 2163.684092] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2163.688465] Lustre: Skipped 1 previous similar message [ 2164.753234] Lustre: server umount lustre-OST0000 complete [ 2168.165451] Lustre: server umount lustre-MDT0000 complete [ 2168.800613] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2168.800819] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2168.801401] LustreError: 120096:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2168.801409] LustreError: 120096:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 2168.821659] Lustre: Skipped 1 previous similar message [ 2169.587395] LustreError: 121720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773104000 with bad export cookie 765695589222633676 [ 2169.591599] LustreError: 121720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2169.591614] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2169.710355] Lustre: server umount lustre-MDT0001 complete [ 2173.597669] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 2174.746807] Key type lgssc unregistered [ 2174.896359] LNet: 123354:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2174.900130] LNetError: 123354:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2174.909528] LNet: Removed LNI 192.168.202.131@tcp [ 2175.238626] Key type .llcrypt unregistered [ 2175.240267] Key type ._llcrypt unregistered [ 2184.932796] Key type ._llcrypt registered [ 2184.934523] Key type .llcrypt registered [ 2184.978682] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_hostid [ 2191.227217] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 2191.717885] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2191.726881] alg: No test for adler32 (adler32-zlib) [ 2192.595126] Lustre: Lustre: Build Version: 2.17.50_236_gea5aa0c [ 2192.692408] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 2194.279167] Key type lgssc registered [ 2194.684623] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2207.886364] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2207.895872] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2209.002682] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2209.018065] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2209.059812] Lustre: lustre-MDT0000: new disk, initializing [ 2209.095948] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2209.104285] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2210.779443] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2215.649033] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2215.697203] Lustre: 126692:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2215.721615] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2215.723667] Lustre: Skipped 1 previous similar message [ 2215.759750] Lustre: lustre-MDT0001: new disk, initializing [ 2215.800792] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2215.821336] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2215.824737] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2217.296315] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2220.785365] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2222.181316] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2224.881880] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2224.985070] Lustre: lustre-OST0000: new disk, initializing [ 2224.987146] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2225.016712] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2225.718586] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2225.722118] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2225.761307] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2227.020460] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2230.328597] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2231.769703] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2231.850872] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2233.326113] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2233.417796] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2235.877281] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2235.878686] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2235.879517] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2235.880652] LustreError: Skipped 1 previous similar message [ 2235.892457] Lustre: Skipped 1 previous similar message [ 2236.244730] Lustre: server umount lustre-OST0000 complete [ 2240.993898] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2240.994924] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2240.999118] Lustre: Skipped 1 previous similar message [ 2241.001276] Lustre: Skipped 1 previous similar message [ 2243.738779] Lustre: server umount lustre-MDT0000 complete [ 2245.053089] LustreError: 126683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773104075 with bad export cookie 3277755780624571877 [ 2245.056109] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2245.057837] LustreError: 126683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2245.176951] Lustre: server umount lustre-MDT0001 complete [ 2248.042300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2248.204286] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2249.483846] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2252.330668] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2252.453203] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2253.722754] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2255.608561] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2256.866478] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2259.331406] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2261.376984] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2263.278631] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2264.931035] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2273.175023] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 8 sec [ 2274.506297] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2274.575736] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2278.057617] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2278.128573] Lustre: lustre-OST0001: new disk, initializing [ 2278.130595] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2278.162712] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2278.166144] Lustre: Skipped 1 previous similar message [ 2280.121534] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2283.333525] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2283.516820] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2283.519932] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2283.545678] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2284.419882] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 2285.669660] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2285.740805] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2286.942055] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8de904bd0000.ost_server_uuid 50 [ 2287.465413] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8de904bd0000.ost_server_uuid in FULL state after 0 sec [ 2293.728199] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2293.732540] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2293.733445] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2293.736217] Lustre: Skipped 1 previous similar message [ 2293.739329] Lustre: Skipped 2 previous similar messages [ 2295.420961] Lustre: server umount lustre-OST0001 complete [ 2298.336652] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2298.336694] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2298.339365] LustreError: Skipped 2 previous similar messages [ 2298.339961] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2298.345264] Lustre: Skipped 1 previous similar message [ 2298.849247] LustreError: 132089:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2298.854332] LustreError: 132089:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 2302.728905] Lustre: server umount lustre-OST0000 complete [ 2303.969504] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2303.969599] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2303.970310] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2303.970318] Lustre: Skipped 1 previous similar message [ 2303.979564] Lustre: Skipped 1 previous similar message [ 2314.208316] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2314.210472] Lustre: Skipped 4 previous similar messages [ 2318.303156] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2318.372757] Lustre: server umount lustre-MDT0000 complete [ 2319.328105] LustreError: 130498:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2319.333649] LustreError: 130498:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 2319.593233] LustreError: 133553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773104150 with bad export cookie 3277755780624572745 [ 2319.598075] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2319.708499] Lustre: server umount lustre-MDT0001 complete [ 2325.639610] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 2326.577298] Key type lgssc unregistered [ 2326.692342] LNet: 136395:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2326.694495] LNetError: 136395:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2326.704283] LNet: Removed LNI 192.168.202.131@tcp [ 2326.962085] Key type .llcrypt unregistered [ 2326.963538] Key type ._llcrypt unregistered [ 2339.110370] Key type ._llcrypt registered [ 2339.111355] Key type .llcrypt registered [ 2339.145986] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_hostid [ 2345.055307] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 2345.408629] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2345.441876] alg: No test for adler32 (adler32-zlib) [ 2346.287769] Lustre: Lustre: Build Version: 2.17.50_236_gea5aa0c [ 2346.363843] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 2347.943156] Key type lgssc registered [ 2348.281682] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2359.449148] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2359.455281] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2360.543113] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2360.571488] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2360.608270] Lustre: lustre-MDT0000: new disk, initializing [ 2360.633356] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2360.640079] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2361.905630] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2365.998855] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2366.043255] Lustre: 140486:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2366.065091] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2366.066952] Lustre: Skipped 1 previous similar message [ 2366.102370] Lustre: lustre-MDT0001: new disk, initializing [ 2366.137118] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2366.149920] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2366.155494] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2367.438792] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2370.477996] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2371.674092] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2373.857990] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2373.953046] Lustre: lustre-OST0000: new disk, initializing [ 2373.954843] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2373.982405] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2375.776132] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2378.794362] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2381.812506] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2381.816971] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2381.844023] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2386.068565] Lustre: Failing over lustre-OST0000 [ 2386.132086] Lustre: server umount lustre-OST0000 complete [ 2398.814694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2398.910589] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2398.914407] Lustre: Skipped 1 previous similar message [ 2398.917824] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2398.923879] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2400.627930] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2400.630608] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2401.783683] Lustre: Failing over lustre-MDT0000 [ 2401.870085] Lustre: server umount lustre-MDT0000 complete [ 2402.271696] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2402.273940] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2402.847140] Lustre: 138715:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773104217/real 1773104217] req@0000000039233ee2 x1859234484480640/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1773104233 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2404.321360] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2407.391093] Lustre: 138713:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773104222/real 1773104222] req@00000000b1124258 x1859234484482432/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1773104238 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2407.401315] Lustre: 138713:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2409.440172] LustreError: 141280:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2409.445008] LustreError: 141280:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 8 previous similar messages [ 2410.861357] LustreError: 141829:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2412.511151] Lustre: 138714:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773104227/real 1773104227] req@0000000019e353b6 x1859234484483584/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1773104243 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2414.495567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2414.532406] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2414.536747] Lustre: 140489:0:(mgc_request.c:1917:mgc_process_log()) MGC192.168.202.131@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 2414.559860] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [ 2414.562350] Lustre: Skipped 2 previous similar messages [ 2414.597165] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 2414.613172] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2414.613635] Lustre: lustre-OST0000: Recovery over after 0:14, of 3 clients 3 recovered and 0 were evicted. [ 2414.618549] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2414.637462] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2415.763596] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2415.981970] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2419.685725] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2419.687711] Lustre: Skipped 1 previous similar message [ 2419.692841] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2419.710793] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2424.802592] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2424.802787] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2424.803756] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2424.805317] LustreError: Skipped 1 previous similar message [ 2424.812714] Lustre: Skipped 2 previous similar messages [ 2426.699118] Lustre: server umount lustre-OST0000 complete [ 2429.920599] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2429.921181] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2429.921416] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2429.921420] Lustre: Skipped 1 previous similar message [ 2429.924285] Lustre: Skipped 1 previous similar message [ 2435.039913] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2435.041801] Lustre: Skipped 2 previous similar messages [ 2440.159982] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2440.161860] Lustre: Skipped 1 previous similar message [ 2442.207206] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2442.266530] Lustre: server umount lustre-MDT0000 complete [ 2443.535136] LustreError: 140479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773104274 with bad export cookie 16356217257020369579 [ 2443.537945] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2443.540142] LustreError: 140479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2443.660281] Lustre: server umount lustre-MDT0001 complete [ 2447.233354] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 2448.271258] Key type lgssc unregistered [ 2448.403336] LNet: 145713:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2448.406188] LNetError: 145713:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2448.415341] LNet: Removed LNI 192.168.202.131@tcp [ 2448.731773] Key type .llcrypt unregistered [ 2448.733094] Key type ._llcrypt unregistered [ 2468.978402] Key type ._llcrypt registered [ 2468.979438] Key type .llcrypt registered [ 2469.019153] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 2469.379024] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2469.413064] alg: No test for adler32 (adler32-zlib) [ 2470.265123] Lustre: Lustre: Build Version: 2.17.50_236_gea5aa0c [ 2470.351664] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 2471.935175] Key type lgssc registered [ 2472.331738] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2474.912615] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2474.917546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2476.054647] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2477.223908] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2480.229966] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2482.139621] LustreError: 147243:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2482.142485] LustreError: 147243:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2482.144947] LustreError: 147243:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 6, retries 0, failed: rc = -5 [ 2482.252213] Lustre: server umount lustre-MDT0000 complete [ 2487.129271] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_hostid [ 2490.481752] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 2502.440540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2502.530221] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2502.541075] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2502.575902] Lustre: lustre-MDT0000: new disk, initializing [ 2502.599032] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2502.605338] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2503.813714] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2507.671776] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2507.707514] Lustre: 150393:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2507.728186] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2507.729854] Lustre: Skipped 1 previous similar message [ 2507.764974] Lustre: lustre-MDT0001: new disk, initializing [ 2507.794621] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2507.806091] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2507.809433] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2508.994501] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2511.971487] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2513.178597] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2515.346538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2515.447154] Lustre: lustre-OST0000: new disk, initializing [ 2515.449525] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2515.477244] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2517.262572] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2520.053557] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2520.057338] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2520.098368] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2520.239777] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2522.053106] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 2522.657916] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 20:59:13 (1773104353) [ 2589.663491] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2589.666292] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2589.670051] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2591.711433] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2591.711565] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2591.713701] Lustre: Skipped 1 previous similar message [ 2591.718747] Lustre: Skipped 1 previous similar message [ 2593.758714] Lustre: server umount lustre-MDT0000 complete [ 2594.965530] LustreError: 153782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773104425 with bad export cookie 15111763131958647923 [ 2594.966475] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2594.968755] LustreError: 153782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2596.831394] LustreError: 150397:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2596.831443] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2596.836210] LustreError: 150397:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 2596.839353] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2596.843663] Lustre: Skipped 1 previous similar message [ 2599.839515] LustreError: 153767:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2599.839523] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2601.115424] Lustre: server umount lustre-MDT0001 complete [ 2608.460924] Lustre: server umount lustre-OST0000 complete [ 2611.268986] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_hostid [ 2614.145830] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 2625.018924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2625.106861] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2625.117031] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2625.154140] Lustre: lustre-MDT0000: new disk, initializing [ 2625.177697] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2625.184272] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2626.513930] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2630.628365] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2630.669797] Lustre: 156608:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2630.691168] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2630.693412] Lustre: Skipped 1 previous similar message [ 2630.732532] Lustre: lustre-MDT0001: new disk, initializing [ 2630.776603] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2630.780854] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2632.016277] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2634.966385] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2636.189857] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2638.310599] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2638.422526] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2638.424319] Lustre: Skipped 1 previous similar message [ 2638.680401] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2638.711677] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2640.195834] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2643.099902] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2644.378107] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2644.446868] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2645.670860] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2645.734254] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2648.394617] Lustre: server umount lustre-OST0000 complete [ 2651.615783] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2651.618032] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2651.621790] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2660.346207] Lustre: server umount lustre-MDT0000 complete [ 2661.554138] LustreError: 156599:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773104492 with bad export cookie 15111763131958806739 [ 2661.556984] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2661.558192] LustreError: 156599:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2672.144561] Lustre: server umount lustre-MDT0001 complete [ 2674.788721] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 21:01:45 (1773104505) [ 2679.712393] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_hostid [ 2682.696581] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 2693.785267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2693.869108] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2693.881108] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2693.882915] Lustre: Skipped 1 previous similar message [ 2693.916567] Lustre: lustre-MDT0000: new disk, initializing [ 2693.917942] Lustre: Skipped 1 previous similar message [ 2693.937042] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2693.941885] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2693.945203] Lustre: Skipped 1 previous similar message [ 2695.095579] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2698.927964] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2698.968492] Lustre: 162933:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2699.063519] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2700.259990] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2703.190827] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2704.337831] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2706.406258] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2708.183057] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2711.161382] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2712.060443] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2712.063172] Lustre: Skipped 1 previous similar message [ 2712.065429] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2712.097099] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2722.273793] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2722.274132] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2722.275826] LustreError: Skipped 1 previous similar message [ 2722.280327] Lustre: Skipped 1 previous similar message [ 2722.280956] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2722.285338] Lustre: Skipped 5 previous similar messages [ 2724.431908] Lustre: server umount lustre-OST0000 complete [ 2731.800422] Lustre: server umount lustre-MDT0000 complete [ 2732.512905] LustreError: 162943:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2732.517540] LustreError: 162943:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 2733.056237] LustreError: 162924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773104563 with bad export cookie 15111763131958807607 [ 2733.059238] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2733.059724] LustreError: 162924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2733.184582] Lustre: server umount lustre-MDT0001 complete [ 2735.880573] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 21:02:46 (1773104566) [ 2740.868174] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_hostid [ 2744.028771] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 2755.575646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2755.665955] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2755.679210] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2755.681173] Lustre: Skipped 3 previous similar messages [ 2755.718713] Lustre: lustre-MDT0000: new disk, initializing [ 2755.721280] Lustre: Skipped 2 previous similar messages [ 2755.745494] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2755.747510] Lustre: Skipped 2 previous similar messages [ 2755.752746] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2757.027972] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2761.132528] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2761.180343] Lustre: 168852:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2761.282478] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2762.612233] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2765.640856] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2766.752214] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2768.948095] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2770.912051] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2773.974875] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2776.600764] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2781.666427] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2781.666474] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2781.670860] Lustre: Skipped 3 previous similar messages [ 2781.674173] LustreError: Skipped 1 previous similar message [ 2781.676660] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2781.680081] Lustre: Skipped 3 previous similar messages [ 2783.321380] Lustre: server umount lustre-OST0000 complete [ 2786.646904] Lustre: server umount lustre-MDT0000 complete [ 2786.785339] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2786.785706] LustreError: 169631:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2786.791815] LustreError: 169631:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 2787.879276] LustreError: 168844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773104618 with bad export cookie 15111763131958808692 [ 2787.882508] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2787.882737] LustreError: 168844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2788.001767] Lustre: server umount lustre-MDT0001 complete [ 2791.626412] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 2792.630466] Key type lgssc unregistered [ 2792.754306] LNet: 172324:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2792.756738] LNetError: 172324:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2792.766294] LNet: Removed LNI 192.168.202.131@tcp [ 2793.043985] Key type .llcrypt unregistered [ 2793.045060] Key type ._llcrypt unregistered [ 2802.279084] Key type ._llcrypt registered [ 2802.280039] Key type .llcrypt registered [ 2802.315913] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 2802.621622] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2802.658588] alg: No test for adler32 (adler32-zlib) [ 2803.508694] Lustre: Lustre: Build Version: 2.17.50_236_gea5aa0c [ 2803.591939] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 2805.167137] Key type lgssc registered [ 2805.501906] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2807.944551] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2807.949968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2809.103318] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2810.323236] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2813.088098] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2813.219979] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2814.484937] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2816.274220] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2817.457428] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2819.667631] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2819.783689] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2821.590234] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2823.331082] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2835.174958] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2850.786122] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2850.786503] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2850.787203] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2850.790524] Lustre: Skipped 1 previous similar message [ 2855.904497] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2855.907394] Lustre: Skipped 2 previous similar messages [ 2856.525828] Lustre: server umount lustre-OST0000 complete [ 2859.487748] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2859.489729] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2859.493324] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2861.024943] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2863.828304] Lustre: server umount lustre-MDT0000 complete [ 2865.047989] LustreError: 173681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773104695 with bad export cookie 6748504801767053570 [ 2865.051059] LustreError: 173681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2865.051226] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2865.163860] Lustre: server umount lustre-MDT0001 complete [ 2868.858790] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 2869.874367] Key type lgssc unregistered [ 2869.996380] LNet: 176900:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2869.998875] LNetError: 176900:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2870.011304] LNet: Removed LNI 192.168.202.131@tcp [ 2870.289203] Key type .llcrypt unregistered [ 2870.290406] Key type ._llcrypt unregistered [ 2879.168674] Key type ._llcrypt registered [ 2879.169726] Key type .llcrypt registered [ 2879.205103] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 2879.500438] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2879.508441] alg: No test for adler32 (adler32-zlib) [ 2880.363613] Lustre: Lustre: Build Version: 2.17.50_236_gea5aa0c [ 2880.444991] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 2882.023165] Key type lgssc registered [ 2882.408360] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2885.111421] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2885.117965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2886.298312] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2887.547583] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2890.219468] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2890.337753] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2891.679341] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2893.483128] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2894.705157] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2896.875971] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2896.993660] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2898.767350] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2900.551578] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2911.333359] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 2920.928025] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2920.930347] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2920.934714] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2921.953809] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2921.954164] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2921.954211] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2921.954216] Lustre: Skipped 1 previous similar message [ 2924.556234] Lustre: server umount lustre-OST0000 complete [ 2926.369846] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8de9072f8000.ost_server_uuid 50 [ 2926.899936] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8de9072f8000.ost_server_uuid in DISCONN state after 0 sec [ 2927.442091] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 2941.408145] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2941.410527] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2941.414458] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2941.416168] Lustre: Skipped 1 previous similar message [ 2943.573453] Lustre: server umount lustre-MDT0000 complete [ 2944.853322] LustreError: 178991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773104775 with bad export cookie 15546208361696144448 [ 2944.856489] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2944.857478] LustreError: 178991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2944.972809] Lustre: server umount lustre-MDT0001 complete [ 2947.840964] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 21:06:18 (1773104778) [ 2950.078730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2950.233739] LustreError: 181806:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2950.241347] LustreError: 181806:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 2950.258397] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2951.556251] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2954.394642] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2954.512167] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2955.767605] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2957.482735] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2958.639644] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2960.822929] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2962.697212] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2964.358790] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2966.558873] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2966.637223] Lustre: lustre-OST0001: new disk, initializing [ 2966.639557] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2966.669931] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2966.671981] Lustre: Skipped 1 previous similar message [ 2968.501669] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2971.554417] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2974.892120] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 2975.242042] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2975.245537] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2975.286824] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2980.326296] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2980.327605] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2980.330233] Lustre: Skipped 1 previous similar message [ 2980.334615] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2980.337059] Lustre: Skipped 3 previous similar messages [ 2983.929905] Lustre: server umount lustre-OST0000 complete [ 2985.444055] LustreError: 183401:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2985.448585] LustreError: 183401:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 2985.763336] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2985.827753] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 2987.107316] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2987.170173] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 2987.375035] LustreError: 183400:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2990.563907] LustreError: 183401:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2990.569568] LustreError: 183401:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 2995.684729] LustreError: 183398:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2995.690068] LustreError: 183398:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 3000.803108] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3000.804538] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3000.805190] LustreError: Skipped 2 previous similar messages [ 3000.806114] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3000.808656] Lustre: Skipped 1 previous similar message [ 3000.813182] Lustre: Skipped 2 previous similar messages [ 3003.283762] Lustre: server umount lustre-OST0001 complete [ 3010.725332] Lustre: server umount lustre-MDT0000 complete [ 3011.040361] LustreError: 181807:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3011.044584] LustreError: 181807:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 3011.997609] LustreError: 181788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773104842 with bad export cookie 15546208361696145505 [ 3012.004339] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3012.122185] Lustre: server umount lustre-MDT0001 complete [ 3019.982918] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 21:07:30 (1773104850) [ 3022.159469] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3022.252975] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3022.264246] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3022.336409] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3023.665255] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3026.460296] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3026.492280] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3026.506201] Lustre: 187575:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3027.963547] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3029.688541] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3030.865341] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3033.115942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3033.179922] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3035.066279] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3035.160029] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 3036.910287] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3039.164516] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3039.191704] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3041.124915] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3042.922982] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3044.323515] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3044.325081] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3044.327099] Lustre: Skipped 3 previous similar messages [ 3044.329161] Lustre: Skipped 2 previous similar messages [ 3044.869255] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 3058.143110] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3058.174733] Lustre: server umount lustre-OST0000 complete [ 3058.544734] LustreError: 189195:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3070.434434] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3070.436763] LustreError: Skipped 1 previous similar message [ 3070.437896] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3070.441602] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3070.443119] Lustre: Skipped 7 previous similar messages [ 3074.827313] Lustre: server umount lustre-OST0001 complete [ 3090.399101] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3090.472767] Lustre: server umount lustre-MDT0000 complete [ 3090.912168] LustreError: 187585:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3090.917240] LustreError: 187585:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 8 previous similar messages [ 3091.703067] LustreError: 188304:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773104922 with bad export cookie 15546208361696146891 [ 3091.706647] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3091.706904] LustreError: 188304:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 3091.826766] Lustre: server umount lustre-MDT0001 complete [ 3099.446808] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 21:08:50 (1773104930) [ 3104.581449] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_hostid [ 3107.649183] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 3119.316498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3119.402022] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3119.412173] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3119.450355] Lustre: lustre-MDT0000: new disk, initializing [ 3119.474666] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3119.476867] Lustre: Skipped 3 previous similar messages [ 3119.482950] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3120.718929] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3124.747892] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3124.785531] Lustre: 195148:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3124.807248] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3124.809410] Lustre: Skipped 1 previous similar message [ 3124.850935] Lustre: lustre-MDT0001: new disk, initializing [ 3124.887661] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3124.891743] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3126.113041] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3129.102828] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3130.254816] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3132.385184] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3132.492594] Lustre: lustre-OST0000: new disk, initializing [ 3132.494605] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3134.278584] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3134.901921] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3134.904822] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3134.936910] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3137.237165] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3138.552310] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3138.617551] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3139.857672] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3139.924297] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3145.187984] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3145.188326] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3145.189972] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3145.189980] Lustre: Skipped 5 previous similar messages [ 3145.193023] LustreError: Skipped 3 previous similar messages [ 3145.205012] Lustre: Skipped 4 previous similar messages [ 3146.635674] Lustre: server umount lustre-OST0000 complete [ 3154.009442] Lustre: server umount lustre-MDT0000 complete [ 3155.242096] LustreError: 195142:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773104985 with bad export cookie 15546208361696148634 [ 3155.244378] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3155.245521] LustreError: 195142:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3155.363380] Lustre: server umount lustre-MDT0001 complete [ 3158.111349] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3158.259886] LustreError: 198954:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3159.565515] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3162.369756] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3163.764298] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3165.506608] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3166.721269] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3168.976657] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3171.015133] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3172.783854] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3174.177760] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3183.430813] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 9 sec [ 3184.837167] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3184.923118] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3187.684855] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3191.708280] Lustre: server umount lustre-OST0000 complete [ 3193.635222] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3194.982440] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3218.536907] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3218.650383] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3218.653564] Lustre: Skipped 5 previous similar messages [ 3220.675437] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3222.520955] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3223.107198] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3226.186864] Lustre: server umount lustre-OST0000 complete [ 3228.513135] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3228.513798] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3228.518913] Lustre: Skipped 5 previous similar messages [ 3228.520859] Lustre: Skipped 5 previous similar messages [ 3229.592988] Lustre: server umount lustre-MDT0000 complete [ 3230.865078] LustreError: 198937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773105061 with bad export cookie 15546208361696149502 [ 3230.867507] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3230.868819] LustreError: 198937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3230.990267] Lustre: server umount lustre-MDT0001 complete [ 3233.937427] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 21:11:04 (1773105064) [ 3236.104734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3237.520855] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3240.347165] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3241.818856] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3243.591128] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3244.852307] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3247.231694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3249.166090] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3250.944220] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3252.433077] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3262.711707] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 10 sec [ 3264.095963] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3264.174724] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3266.509242] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3266.579874] Lustre: lustre-OST0001: new disk, initializing [ 3266.581831] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3268.545494] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3271.669574] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3273.096316] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3275.776390] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3275.779035] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3275.809299] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3276.224912] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 3277.571231] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3277.643372] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3280.867512] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3280.871176] LustreError: Skipped 1 previous similar message [ 3284.352517] Lustre: server umount lustre-OST0001 complete [ 3286.321084] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3286.395648] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3287.738269] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3287.810992] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3287.933951] LustreError: 206050:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3287.941044] LustreError: 206050:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 3311.473300] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3313.433122] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3315.224782] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3315.757541] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3322.489319] Lustre: server umount lustre-OST0001 complete [ 3340.687714] Lustre: server umount lustre-OST0000 complete [ 3348.308756] Lustre: server umount lustre-MDT0000 complete [ 3349.428462] LustreError: 210386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773105180 with bad export cookie 15546208361696150419 [ 3349.430906] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3349.431967] LustreError: 210386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3355.635336] Lustre: server umount lustre-MDT0001 complete [ 3363.792196] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 21:13:14 (1773105194) [ 3366.058334] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3366.160065] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3366.171020] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3366.246554] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3366.249577] Lustre: Skipped 5 previous similar messages [ 3367.609804] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3370.596334] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3370.634716] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3370.648494] Lustre: 212483:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3372.160190] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3374.054189] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3375.341159] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3377.711366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3377.783544] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3379.878259] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3381.782260] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3385.629042] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3387.595377] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3389.462064] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3391.040659] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec [ 3392.423396] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3392.501484] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3393.775841] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3393.850685] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3395.126700] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8de905464000.ost_server_uuid 50 [ 3395.634769] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8de905464000.ost_server_uuid in FULL state after 0 sec [ 3395.978100] Lustre: Permanently deactivating lustre-OST0001 [ 3395.984187] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3396.829513] Lustre: Permanently reactivating lustre-OST0001 [ 3402.208969] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3402.213618] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3402.214789] Lustre: Skipped 7 previous similar messages [ 3402.216216] Lustre: Skipped 16 previous similar messages [ 3403.966693] Lustre: server umount lustre-OST0001 complete [ 3411.347366] Lustre: server umount lustre-OST0000 complete [ 3417.055963] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3417.058125] LustreError: Skipped 2 previous similar messages [ 3418.783168] Lustre: server umount lustre-MDT0000 complete [ 3420.098226] LustreError: 212474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773105250 with bad export cookie 15546208361696151679 [ 3420.100871] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3420.103446] LustreError: 212474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3420.226132] Lustre: server umount lustre-MDT0001 complete [ 3428.215360] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 21:14:19 (1773105259) [ 3430.769989] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3430.871332] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3430.882462] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3430.887503] Lustre: Skipped 5 previous similar messages [ 3432.292488] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3435.120079] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3435.153968] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3435.156915] Lustre: Skipped 1 previous similar message [ 3435.168124] Lustre: 218580:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3436.580979] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3438.312405] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3439.594880] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3441.952160] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3442.034837] Lustre: Permanently deactivating lustre-OST0000 [ 3442.040398] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 3444.087019] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3445.934393] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3447.286066] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3447.290651] LustreError: 220948:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0001: cannot cleanup orphans: rc = -108 [ 3448.217552] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3450.250465] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3452.089438] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3455.514511] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 3457.851867] Lustre: Permanently reactivating lustre-OST0000 [ 3464.690124] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 3464.693588] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3464.696608] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3464.707573] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 3464.714457] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3468.867494] Lustre: server umount lustre-OST0001 complete [ 3476.174719] Lustre: server umount lustre-OST0000 complete [ 3483.485109] Lustre: server umount lustre-MDT0000 complete [ 3484.771248] LustreError: 218571:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773105315 with bad export cookie 15546208361696153324 [ 3484.774850] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3484.776105] LustreError: 218571:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3484.915158] Lustre: server umount lustre-MDT0001 complete [ 3488.691812] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 3489.823892] Key type lgssc unregistered [ 3489.943428] LNet: 222956:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3489.947258] LNetError: 222956:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3489.954341] LNet: Removed LNI 192.168.202.131@tcp [ 3490.261469] Key type .llcrypt unregistered [ 3490.262710] Key type ._llcrypt unregistered [ 3499.047708] Key type ._llcrypt registered [ 3499.049161] Key type .llcrypt registered [ 3499.086700] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 3499.361913] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3499.384546] alg: No test for adler32 (adler32-zlib) [ 3500.246508] Lustre: Lustre: Build Version: 2.17.50_236_gea5aa0c [ 3500.328921] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 3501.919184] Key type lgssc registered [ 3502.299618] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3505.728285] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3505.735939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3506.910848] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3508.272124] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3511.144413] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3511.191440] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3511.202567] Lustre: 224368:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3511.206945] Lustre: Permanently deactivating lustre-MDT0001 [ 3511.211889] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3511.322255] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3512.624573] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3514.518582] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3515.781679] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3516.403437] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3518.041459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3518.158802] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3520.050417] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3521.248541] LustreError: 225996:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3521.253290] LustreError: 225996:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 3521.889676] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3524.213226] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3524.290078] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3524.343363] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 3526.270455] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3528.111203] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3532.520500] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 3532.520574] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 3534.124200] Lustre: Permanently reactivating lustre-MDT0001 [ 3534.130491] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 3534.132960] Lustre: Skipped 2 previous similar messages [ 3534.396098] Lustre: lustre-MDT0001: Client 9e3250f4-21f8-4d9c-94a0-74d2708d3637 (at 192.168.202.31@tcp) reconnecting [ 3536.380863] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3536.931418] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3538.606110] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 3538.685979] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 3540.029611] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 3540.111129] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 3540.486977] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3540.494525] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 3540.497153] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 3540.504635] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3540.958560] Lustre: Permanently deactivating lustre-MDT0001 [ 3540.964922] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3540.968179] Lustre: Skipped 2 previous similar messages [ 3547.135769] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3550.178557] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3550.182736] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3550.184647] Lustre: Skipped 2 previous similar messages [ 3550.187477] Lustre: Skipped 1 previous similar message [ 3555.296871] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3555.298778] Lustre: Skipped 5 previous similar messages [ 3555.803809] Lustre: server umount lustre-MDT0000 complete [ 3557.026218] LustreError: 224362:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773105387 with bad export cookie 3667707476306064299 [ 3557.030373] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3557.030981] LustreError: 224362:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3557.146943] Lustre: server umount lustre-MDT0001 complete [ 3564.475396] Lustre: server umount lustre-OST0000 complete [ 3568.313611] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 21:16:39 (1773105399) [ 3571.995896] Lustre: server umount lustre-OST0001 complete [ 3573.916687] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_hostid [ 3576.909955] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 3588.054763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3588.135978] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3588.138552] Lustre: Skipped 2 previous similar messages [ 3588.146357] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3588.178952] Lustre: lustre-MDT0000: new disk, initializing [ 3588.199859] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3588.206036] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3589.519652] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3593.444953] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3593.483572] Lustre: 232019:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3593.504882] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3593.507032] Lustre: Skipped 1 previous similar message [ 3593.544270] Lustre: lustre-MDT0001: new disk, initializing [ 3593.587119] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3593.590548] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3594.813689] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3597.734051] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3598.849977] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3601.014630] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3601.129239] Lustre: lustre-OST0000: new disk, initializing [ 3601.130840] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3601.158727] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3601.162033] Lustre: Skipped 1 previous similar message [ 3602.936929] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3602.940300] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3602.949588] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3602.986887] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3605.932376] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3606.573650] LustreError: 232801:0:(mdt_handler.c:3251:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3610.730857] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3610.793776] Lustre: lustre-OST0001: new disk, initializing [ 3610.796533] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3612.578832] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3615.594563] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3616.132284] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3616.671133] LustreError: 232801:0:(mdt_handler.c:3251:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3616.678859] LustreError: 232024:0:(mdt_handler.c:3251:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3620.868701] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3620.872667] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3620.907078] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3626.767114] LustreError: 232024:0:(mdt_handler.c:3251:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3631.074954] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3631.075076] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3631.077969] LustreError: Skipped 1 previous similar message [ 3631.082354] Lustre: Skipped 1 previous similar message [ 3631.083033] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3631.087580] Lustre: Skipped 1 previous similar message [ 3633.660632] Lustre: server umount lustre-OST0001 complete [ 3636.195838] LustreError: 233744:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3636.203294] LustreError: 233744:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 3641.314981] LustreError: 233744:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3641.318467] LustreError: 233744:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 3646.435723] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3646.436745] LustreError: 233743:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3646.438923] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3646.440771] Lustre: Skipped 2 previous similar messages [ 3646.445313] LustreError: 233743:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 3651.554973] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3651.557132] Lustre: Skipped 2 previous similar messages [ 3651.557920] LustreError: 233744:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3651.562386] LustreError: 233744:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 3651.732185] Lustre: server umount lustre-OST0000 complete [ 3655.136136] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3655.138440] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3659.163641] Lustre: server umount lustre-MDT0000 complete [ 3660.391764] LustreError: 232009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773105491 with bad export cookie 3667707476306066868 [ 3660.396606] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3660.400645] LustreError: 232009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3666.610469] Lustre: server umount lustre-MDT0001 complete [ 3670.344760] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 3671.399172] Key type lgssc unregistered [ 3671.538350] LNet: 236679:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3671.540837] LNetError: 236679:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3671.549702] LNet: Removed LNI 192.168.202.131@tcp [ 3671.845277] Key type .llcrypt unregistered [ 3671.846996] Key type ._llcrypt unregistered [ 3686.291741] Key type ._llcrypt registered [ 3686.292834] Key type .llcrypt registered [ 3686.327163] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 3686.748772] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3686.763731] alg: No test for adler32 (adler32-zlib) [ 3687.607704] Lustre: Lustre: Build Version: 2.17.50_236_gea5aa0c [ 3687.688776] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 3689.271157] Key type lgssc registered [ 3689.608273] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3692.122521] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3692.129812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3693.219739] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3693.227622] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3693.288252] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3694.604299] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3697.504682] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3697.537289] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3697.549393] Lustre: 238847:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3697.661802] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3699.046836] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3700.915205] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3702.246945] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3704.605025] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3704.672937] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3704.735991] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3706.656989] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3708.537505] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3709.469968] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 3755.490283] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3755.490330] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3755.492374] LustreError: Skipped 1 previous similar message [ 3755.492959] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3755.498061] Lustre: Skipped 1 previous similar message [ 3759.134957] Lustre: server umount lustre-OST0000 complete [ 3760.596550] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 3764.731102] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3764.843717] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3766.657695] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3768.386925] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3776.100718] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3776.105544] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3776.105748] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3776.112688] Lustre: Skipped 1 previous similar message [ 3802.082066] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3802.082457] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3802.084813] LustreError: Skipped 1 previous similar message [ 3802.090815] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3802.093610] Lustre: Skipped 2 previous similar messages [ 3807.202659] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3807.204825] Lustre: Skipped 1 previous similar message [ 3807.766350] Lustre: server umount lustre-OST0000 complete [ 3810.271899] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3810.274340] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3810.277846] Lustre: Skipped 1 previous similar message [ 3810.279598] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3812.320975] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3815.126790] Lustre: server umount lustre-MDT0000 complete [ 3816.275461] LustreError: 238839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773105646 with bad export cookie 1467976637235107299 [ 3816.278748] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3816.281556] LustreError: 238839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3816.394145] Lustre: server umount lustre-MDT0001 complete [ 3819.745779] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 3820.633558] Key type lgssc unregistered [ 3820.742314] LNet: 243791:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3820.744262] LNetError: 243791:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3820.752247] LNet: Removed LNI 192.168.202.131@tcp [ 3821.008376] Key type .llcrypt unregistered [ 3821.009838] Key type ._llcrypt unregistered [ 3829.834263] Key type ._llcrypt registered [ 3829.835500] Key type .llcrypt registered [ 3829.867074] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 3830.157549] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3830.172052] alg: No test for adler32 (adler32-zlib) [ 3831.012898] Lustre: Lustre: Build Version: 2.17.50_236_gea5aa0c [ 3831.089330] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 3832.671126] Key type lgssc registered [ 3832.994977] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3835.344151] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3835.349936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3836.501502] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3837.707707] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3840.458358] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3840.599450] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3841.830933] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3843.510371] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3844.607922] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3846.768763] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3846.883633] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3848.684192] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3850.311835] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3862.245883] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 3876.831956] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3876.834033] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3876.837445] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3877.860565] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3877.862373] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3877.863073] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3879.181305] Lustre: server umount lustre-OST0000 complete [ 3881.440104] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3881.442204] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3881.445808] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3881.447356] Lustre: Skipped 1 previous similar message [ 3886.483785] Lustre: server umount lustre-MDT0000 complete [ 3887.627172] LustreError: 246779:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773105718 with bad export cookie 14414340439411547805 [ 3887.630192] LustreError: 246779:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3887.630227] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3887.742914] Lustre: server umount lustre-MDT0001 complete [ 3891.043348] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 3891.915783] Key type lgssc unregistered [ 3892.026374] LNet: 249135:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3892.029928] LNetError: 249135:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3892.042310] LNet: Removed LNI 192.168.202.131@tcp [ 3892.293292] Key type .llcrypt unregistered [ 3892.294396] Key type ._llcrypt unregistered [ 3899.195848] Key type ._llcrypt registered [ 3899.196762] Key type .llcrypt registered [ 3899.229166] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 3899.497813] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3899.506299] alg: No test for adler32 (adler32-zlib) [ 3900.343535] Lustre: Lustre: Build Version: 2.17.50_236_gea5aa0c [ 3900.416025] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 3901.991113] Key type lgssc registered [ 3902.302143] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3904.739735] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3904.748853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3905.905242] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3907.188580] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3909.810825] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3909.936082] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3911.109995] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3912.729659] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3913.909163] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3915.962190] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3916.077961] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3917.096629] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 3917.872768] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3919.494423] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3925.473044] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3925.475181] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3925.476876] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3925.481724] Lustre: Skipped 1 previous similar message [ 3925.484387] Lustre: Skipped 1 previous similar message [ 3929.940204] Lustre: server umount lustre-OST0000 complete [ 3935.713575] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3935.713768] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3935.714816] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3935.725264] Lustre: Skipped 1 previous similar message [ 3937.241866] Lustre: server umount lustre-MDT0000 complete [ 3938.392191] LustreError: 252700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773105769 with bad export cookie 2063974010712219322 [ 3938.395182] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3938.397733] LustreError: 252700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3938.516833] Lustre: server umount lustre-MDT0001 complete [ 3942.155963] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 3943.130032] Key type lgssc unregistered [ 3943.262464] LNet: 253635:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3943.266234] LNetError: 253635:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3943.275439] LNet: Removed LNI 192.168.202.131@tcp [ 3943.584372] Key type .llcrypt unregistered [ 3943.585486] Key type ._llcrypt unregistered [ 3952.968204] Key type ._llcrypt registered [ 3952.969294] Key type .llcrypt registered [ 3953.001639] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 3953.387473] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3953.397208] alg: No test for adler32 (adler32-zlib) [ 3954.247303] Lustre: Lustre: Build Version: 2.17.50_236_gea5aa0c [ 3954.324178] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 3955.903178] Key type lgssc registered [ 3956.299537] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3958.951592] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3958.957198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3960.106541] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3961.291119] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3963.817791] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3963.930285] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3965.062593] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3966.610728] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3967.667364] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3969.710068] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3969.815572] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3970.853792] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3971.436419] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3973.075863] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3981.279737] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3981.281130] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3981.283051] LustreError: Skipped 1 previous similar message [ 3981.286599] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3984.352747] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3984.355166] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3984.357326] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3987.023045] Lustre: server umount lustre-OST0000 complete [ 3989.473876] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3989.474154] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3989.474914] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3989.474920] Lustre: Skipped 1 previous similar message [ 3989.482095] Lustre: Skipped 1 previous similar message [ 3994.400966] Lustre: server umount lustre-MDT0000 complete [ 3994.592347] LustreError: 255013:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3994.596702] LustreError: 255013:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 3995.583461] LustreError: 254992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773105826 with bad export cookie 15415423780101694909 [ 3995.585930] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3995.586419] LustreError: 254992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3995.697194] Lustre: server umount lustre-MDT0001 complete [ 3998.790538] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 3999.695163] Key type lgssc unregistered [ 3999.807362] LNet: 258960:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3999.809367] LNetError: 258960:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3999.817299] LNet: Removed LNI 192.168.202.131@tcp [ 4000.077964] Key type .llcrypt unregistered [ 4000.079045] Key type ._llcrypt unregistered [ 4007.039119] Key type ._llcrypt registered [ 4007.040116] Key type .llcrypt registered [ 4007.072969] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 4007.349963] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4007.364607] alg: No test for adler32 (adler32-zlib) [ 4008.203090] Lustre: Lustre: Build Version: 2.17.50_236_gea5aa0c [ 4008.276882] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 4009.855150] Key type lgssc registered [ 4010.155265] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4012.393140] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4012.398414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4013.572802] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4014.706216] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 4017.262277] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4017.384498] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4018.619648] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 4020.375020] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4021.519177] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4023.689625] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4023.809310] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4025.700223] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 4026.920105] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 4027.385738] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4031.968593] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4031.969270] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4031.970696] LustreError: Skipped 1 previous similar message [ 4031.971397] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4031.973847] Lustre: Skipped 1 previous similar message [ 4031.979053] Lustre: Skipped 1 previous similar message [ 4033.738512] Lustre: server umount lustre-OST0000 complete [ 4037.088522] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4037.090043] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4037.092212] Lustre: Skipped 1 previous similar message [ 4037.096637] Lustre: Skipped 1 previous similar message [ 4040.988040] Lustre: server umount lustre-MDT0000 complete [ 4042.207993] LustreError: 260159:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4042.212253] LustreError: 260159:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 4042.305356] LustreError: 260123:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773105872 with bad export cookie 8465386091249232849 [ 4042.308715] LustreError: 260123:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4042.308754] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4042.470224] Lustre: server umount lustre-MDT0001 complete [ 4046.438636] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 4047.697399] Key type lgssc unregistered [ 4047.856756] LNet: 263501:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4047.860141] LNetError: 263501:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4047.867394] LNet: Removed LNI 192.168.202.131@tcp [ 4048.243317] Key type .llcrypt unregistered [ 4048.244902] Key type ._llcrypt unregistered [ 4052.965439] Key type ._llcrypt registered [ 4052.966977] Key type .llcrypt registered [ 4053.019739] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 4060.915843] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_hostid [ 4068.534975] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 4069.066662] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4069.077240] alg: No test for adler32 (adler32-zlib) [ 4069.946566] Lustre: Lustre: Build Version: 2.17.50_236_gea5aa0c [ 4070.038703] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 4071.615131] Key type lgssc registered [ 4071.937784] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4082.640861] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4082.645444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4083.733088] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4083.742976] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4083.774976] Lustre: lustre-MDT0000: new disk, initializing [ 4083.797305] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4083.803673] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4085.063100] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 4089.042409] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4089.083242] Lustre: 266975:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4089.102452] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4089.104082] Lustre: Skipped 1 previous similar message [ 4089.136180] Lustre: lustre-MDT0001: new disk, initializing [ 4089.166169] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4089.179716] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4089.182968] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4090.379557] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 4093.365454] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4094.534723] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4096.616330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4096.713062] Lustre: lustre-OST0000: new disk, initializing [ 4096.715467] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4096.741948] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4098.550831] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 4099.638823] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4099.642991] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4099.674395] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4101.541536] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4102.820618] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4102.893600] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4104.133966] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4104.199988] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4106.899839] Lustre: server umount lustre-OST0000 complete [ 4109.792689] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4109.793159] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4109.796112] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4109.799934] Lustre: Skipped 1 previous similar message [ 4109.804595] Lustre: Skipped 1 previous similar message [ 4114.912430] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4114.915107] Lustre: Skipped 1 previous similar message [ 4118.006860] Lustre: server umount lustre-MDT0000 complete [ 4119.137158] LustreError: 266967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773105949 with bad export cookie 878364271301171295 [ 4119.139851] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4119.140256] LustreError: 266967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4128.785242] Lustre: server umount lustre-MDT0001 complete [ 4131.279992] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 21:26:02 (1773105962) [ 4133.250824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4133.413872] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4134.553108] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 4137.186900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4138.714105] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 4140.404844] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4141.590350] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4143.613978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4143.705807] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4143.708526] Lustre: Skipped 1 previous similar message [ 4145.370537] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 4146.988395] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4167.647810] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4167.649708] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4167.653779] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4169.696329] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4169.698446] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4169.698677] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4169.704272] Lustre: Skipped 1 previous similar message [ 4171.217532] Lustre: server umount lustre-OST0000 complete [ 4173.279839] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4173.281824] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4174.816740] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4174.819747] Lustre: Skipped 2 previous similar messages [ 4178.578857] Lustre: server umount lustre-MDT0000 complete [ 4179.789523] LustreError: 270958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773106010 with bad export cookie 878364271301172163 [ 4179.792564] LustreError: 271696:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000bd2:0x1:0x0]: rc = -5 [ 4179.793188] LustreError: 270958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4179.794100] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4179.920849] Lustre: server umount lustre-MDT0001 complete [ 4183.417759] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 4184.472640] Key type lgssc unregistered [ 4184.605409] LNet: 274177:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4184.608981] LNetError: 274177:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4184.620309] LNet: Removed LNI 192.168.202.131@tcp [ 4184.914534] Key type .llcrypt unregistered [ 4184.915507] Key type ._llcrypt unregistered [ 4193.234334] Key type ._llcrypt registered [ 4193.235202] Key type .llcrypt registered [ 4193.270439] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_hostid [ 4199.275626] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 4199.714343] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4199.731991] alg: No test for adler32 (adler32-zlib) [ 4200.584728] Lustre: Lustre: Build Version: 2.17.50_236_gea5aa0c [ 4200.671443] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 4202.255159] Key type lgssc registered [ 4202.615317] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4216.949616] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4216.954516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4218.041301] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4218.053655] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4218.087310] Lustre: lustre-MDT0000: new disk, initializing [ 4218.112820] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4218.120876] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4219.382395] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 4223.347453] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4223.390818] Lustre: 277942:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4223.414280] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4223.416713] Lustre: Skipped 1 previous similar message [ 4223.449081] Lustre: lustre-MDT0001: new disk, initializing [ 4223.477438] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4223.488732] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4223.492503] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4224.735754] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 4227.700503] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4228.804048] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4230.889453] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4230.988939] Lustre: lustre-OST2710: new disk, initializing [ 4230.991635] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 4231.018599] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 4232.780291] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 4235.700789] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 4236.279907] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 4236.283372] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 4236.318247] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 4237.886174] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4237.942717] Lustre: lustre-OST03e8: new disk, initializing [ 4237.945617] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 4237.974124] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 4239.756131] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 4241.459800] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 4243.986822] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 4244.056177] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4244.477722] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 4244.481384] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 4244.505247] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 4245.276448] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 4245.346542] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4246.658654] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 4246.725969] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4247.974899] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 4248.042592] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4249.055759] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4249.059390] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4249.064992] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4249.569101] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4249.571288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4249.572318] Lustre: Skipped 1 previous similar message [ 4254.688545] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4254.690254] Lustre: Skipped 3 previous similar messages [ 4255.128811] Lustre: server umount lustre-MDT0000 complete [ 4256.286096] LustreError: 277931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773106086 with bad export cookie 947383792855242087 [ 4256.289727] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4256.291179] LustreError: 277931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4256.394334] Lustre: server umount lustre-MDT0001 complete [ 4268.033883] Lustre: server umount lustre-OST2710 complete [ 4279.446435] Lustre: server umount lustre-OST03e8 complete [ 4284.993515] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_hostid [ 4287.778891] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 4298.741229] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4298.836153] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4298.849165] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4298.883976] Lustre: lustre-MDT0000: new disk, initializing [ 4298.908970] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4298.916231] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4300.143607] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 4304.122403] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4304.164892] Lustre: 285655:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4304.262175] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4305.594029] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 4308.575854] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4309.720493] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4311.929704] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4312.039519] Lustre: lustre-OST0000: new disk, initializing [ 4312.040792] Lustre: Skipped 1 previous similar message [ 4312.042519] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4312.044530] Lustre: Skipped 2 previous similar messages [ 4312.074216] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4312.076224] Lustre: Skipped 1 previous similar message [ 4313.922488] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 4317.018323] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4318.400638] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4320.252591] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4320.256257] Lustre: Skipped 1 previous similar message [ 4320.258295] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4320.294749] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4320.509607] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 4321.750060] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4321.816580] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4325.344826] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4325.345587] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4325.347368] LustreError: Skipped 1 previous similar message [ 4325.350676] Lustre: Skipped 1 previous similar message [ 4325.351407] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4325.354725] Lustre: Skipped 3 previous similar messages [ 4328.525338] Lustre: server umount lustre-OST0000 complete [ 4329.951928] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4329.955562] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4329.959876] Lustre: Skipped 1 previous similar message [ 4329.962581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4340.704053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4340.706597] Lustre: Skipped 5 previous similar messages [ 4344.287111] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4344.344886] Lustre: server umount lustre-MDT0000 complete [ 4345.531625] LustreError: 285645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773106176 with bad export cookie 947383792855243753 [ 4345.537783] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4345.669752] Lustre: server umount lustre-MDT0001 complete [ 4348.438273] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 21:29:39 (1773106179) [ 4348.915675] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 4349.447550] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 21:29:40 (1773106180) [ 4350.415805] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 4359.121937] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4359.217337] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4359.224409] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4359.284649] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4360.662644] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 4363.504175] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4363.538678] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4363.550303] Lustre: 291101:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4364.948124] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 4366.649590] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4367.773693] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4383.073995] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4383.138300] Lustre: Denying initial registration attempt from nid 192.168.202.131@tcp, specified as failover [ 4383.141390] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 4383.144692] LustreError: 292710:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -99 [ 4383.147434] LustreError: 292710:0:(tgt_mount.c:2064:server_put_super()) no obd lustre-OST0000 [ 4383.149898] LustreError: 292710:0:(tgt_mount.c:115:server_deregister_mount()) lustre-OST0000 not registered [ 4383.178266] Lustre: server umount lustre-OST0000 complete [ 4383.179660] LustreError: 292710:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 4389.343778] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4389.346117] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4389.349573] Lustre: Skipped 1 previous similar message [ 4389.351254] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4391.443269] Lustre: server umount lustre-MDT0000 complete [ 4392.653210] LustreError: 291092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773106223 with bad export cookie 947383792855244621 [ 4392.657136] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4392.657200] LustreError: 291092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 4392.768965] Lustre: server umount lustre-MDT0001 complete [ 4396.263101] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 4397.309486] Key type lgssc unregistered [ 4397.440406] LNet: 293814:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4397.443098] LNetError: 293814:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4397.451358] LNet: Removed LNI 192.168.202.131@tcp [ 4397.755068] Key type .llcrypt unregistered [ 4397.756164] Key type ._llcrypt unregistered [ 4401.445878] Key type ._llcrypt registered [ 4401.446939] Key type .llcrypt registered [ 4401.500921] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 4401.962693] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4402.008774] alg: No test for adler32 (adler32-zlib) [ 4402.870153] Lustre: Lustre: Build Version: 2.17.50_236_gea5aa0c [ 4402.950746] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 4404.535146] Key type lgssc registered [ 4404.915415] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4439.589599] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 4442.387644] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4442.393118] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4443.475631] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4443.482837] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4443.549673] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4444.756293] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 4447.510349] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4447.550293] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4447.560809] Lustre: 296531:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4447.656887] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4448.928965] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 4450.596270] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4451.839952] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4467.106169] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4467.170350] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4467.238753] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4469.052536] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 4470.766713] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4471.545017] Lustre: server umount lustre-OST0000 complete [ 4473.311873] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4473.314349] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4473.318562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4475.887926] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4475.888191] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4480.992064] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4480.993987] Lustre: Skipped 2 previous similar messages [ 4486.112284] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4486.114156] Lustre: Skipped 1 previous similar message [ 4487.135081] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4487.185964] Lustre: server umount lustre-MDT0000 complete [ 4488.457156] LustreError: 296522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773106319 with bad export cookie 12151902159802425675 [ 4488.459476] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4488.462198] LustreError: 296522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4488.580491] Lustre: server umount lustre-MDT0001 complete [ 4492.119888] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 4493.215501] Key type lgssc unregistered [ 4493.344370] LNet: 299735:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4493.347671] LNetError: 299735:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4493.355322] LNet: Removed LNI 192.168.202.131@tcp [ 4493.660725] Key type .llcrypt unregistered [ 4493.661769] Key type ._llcrypt unregistered [ 4502.271461] Key type ._llcrypt registered [ 4502.272618] Key type .llcrypt registered [ 4502.307121] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 4502.756642] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4502.777528] alg: No test for adler32 (adler32-zlib) [ 4503.624633] Lustre: Lustre: Build Version: 2.17.50_236_gea5aa0c [ 4503.714323] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 4505.295181] Key type lgssc registered [ 4505.704548] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4508.407029] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4508.415052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4509.584207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4510.940765] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 4513.789217] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4513.916326] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4515.214625] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 4517.016784] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4518.230461] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4522.401996] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4522.520835] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4524.390891] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 4526.127467] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4539.872123] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4539.875081] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4539.881329] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4540.896285] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4540.897931] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4542.835151] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.31@tcp (stopping) [ 4542.838985] Lustre: Skipped 2 previous similar messages [ 4545.753521] Lustre: server umount lustre-MDT0000 complete [ 4546.016106] LustreError: 301111:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4546.020593] LustreError: 301111:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 4 previous similar messages [ 4547.001108] LustreError: 301092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773106377 with bad export cookie 5407174583521269566 [ 4547.004908] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4547.949645] LustreError: 301135:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4547.955984] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.31@tcp (stopping) [ 4551.135972] LustreError: 303266:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4551.136262] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4551.146986] Lustre: Skipped 1 previous similar message [ 4552.160146] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4552.162695] Lustre: Skipped 1 previous similar message [ 4557.280051] LustreError: 301106:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4557.284351] LustreError: 301106:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 4561.375168] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4561.457989] Lustre: server umount lustre-MDT0001 complete [ 4563.566603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4566.322254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4571.615617] LustreError: 300421:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000d5de7a72 x1859236747306368/t0(0) o250->MGC192.168.202.131@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4571.737268] LustreError: 302714:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4571.746632] LustreError: 302714:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 4571.763860] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4571.786274] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [ 4573.057962] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 4575.960176] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4576.098842] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4577.407065] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 4579.094823] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4587.492284] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4588.785607] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 9 sec [ 4589.944026] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4590.461950] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4592.611705] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4592.612561] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4592.615632] Lustre: Skipped 1 previous similar message [ 4592.617634] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4592.618393] Lustre: Skipped 4 previous similar messages [ 4597.792911] Lustre: server umount lustre-OST0000 complete [ 4601.824247] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4601.828486] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4605.218404] Lustre: server umount lustre-MDT0000 complete [ 4606.538838] LustreError: 302728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773106437 with bad export cookie 5407174583521276685 [ 4606.541911] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4606.542637] LustreError: 302728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 4606.686860] Lustre: server umount lustre-MDT0001 complete [ 4610.494370] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 4611.500245] Key type lgssc unregistered [ 4611.628362] LNet: 306818:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4611.630561] LNetError: 306818:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4611.644271] LNet: Removed LNI 192.168.202.131@tcp [ 4611.939120] Key type .llcrypt unregistered [ 4611.940227] Key type ._llcrypt unregistered [ 4621.080547] Key type ._llcrypt registered [ 4621.082248] Key type .llcrypt registered [ 4621.122104] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 4621.419517] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4621.458511] alg: No test for adler32 (adler32-zlib) [ 4622.324488] Lustre: Lustre: Build Version: 2.17.50_236_gea5aa0c [ 4622.417658] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 4623.999155] Key type lgssc registered [ 4624.352723] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4626.883229] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4626.889712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4628.046638] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4629.318432] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 4632.091821] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4632.216534] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4633.512359] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 4635.302727] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4636.589848] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4637.612869] LustreError: 308224:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4637.616148] LustreError: 308224:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4637.618891] LustreError: 308224:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 10, retries 0, failed: rc = -5 [ 4637.665468] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4637.665719] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4637.666787] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4637.673342] Lustre: Skipped 1 previous similar message [ 4641.760307] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4641.762282] Lustre: Skipped 2 previous similar messages [ 4643.795952] Lustre: server umount lustre-MDT0000 complete [ 4644.977253] LustreError: 308175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773106475 with bad export cookie 3198896003769411244 [ 4644.980669] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4644.982345] LustreError: 308175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4645.092130] Lustre: server umount lustre-MDT0001 complete [ 4649.699324] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4649.792097] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4649.802973] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4649.875972] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4651.108824] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 4653.837710] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4653.867280] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4653.878190] Lustre: 310556:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4653.979052] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4655.231933] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 4656.923914] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4658.208444] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4662.881872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4662.948230] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 4662.951191] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 4662.954168] LustreError: 312210:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 4662.956654] LustreError: 312210:0:(tgt_mount.c:2064:server_put_super()) no obd lustre-OST0000 [ 4662.958685] LustreError: 312210:0:(tgt_mount.c:115:server_deregister_mount()) lustre-OST0000 not registered [ 4662.979506] Lustre: server umount lustre-OST0000 complete [ 4662.980769] LustreError: 312210:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 4667.746023] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4667.813819] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4667.873073] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4668.357809] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [ 4669.653925] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 4671.371800] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4673.810112] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4673.841239] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4673.869589] Lustre: lustre-OST0001: new disk, initializing [ 4673.871430] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4675.838605] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 4676.044587] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4676.050488] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4676.105948] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4677.732689] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4681.190453] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4681.193499] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4681.196151] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4681.197441] Lustre: Skipped 1 previous similar message [ 4684.863502] Lustre: server umount lustre-OST0001 complete [ 4686.305849] LustreError: 312674:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4686.306064] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4686.307587] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4686.307593] Lustre: Skipped 1 previous similar message [ 4686.310940] LustreError: 312674:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 4686.316679] Lustre: Skipped 1 previous similar message [ 4691.426343] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4691.427365] LustreError: 312677:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4691.428222] Lustre: Skipped 2 previous similar messages [ 4691.438375] LustreError: 312677:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 4692.236879] Lustre: server umount lustre-OST0000 complete [ 4695.008039] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4695.013403] LustreError: Skipped 1 previous similar message [ 4695.015816] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4699.629050] Lustre: server umount lustre-MDT0000 complete [ 4700.871736] LustreError: 311276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773106531 with bad export cookie 3198896003769411762 [ 4700.875338] LustreError: 311276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4700.876220] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4701.007078] Lustre: server umount lustre-MDT0001 complete [ 4704.468057] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 4705.532182] Key type lgssc unregistered [ 4705.664507] LNet: 315463:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4705.666747] LNetError: 315463:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4705.673526] LNet: Removed LNI 192.168.202.131@tcp [ 4705.975124] Key type .llcrypt unregistered [ 4705.976121] Key type ._llcrypt unregistered [ 4736.314826] Key type ._llcrypt registered [ 4736.315970] Key type .llcrypt registered [ 4736.350467] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_hostid [ 4742.438394] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 4743.034533] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4743.052609] alg: No test for adler32 (adler32-zlib) [ 4743.921745] Lustre: Lustre: Build Version: 2.17.50_236_gea5aa0c [ 4744.020241] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 4745.599178] Key type lgssc registered [ 4746.021684] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4757.677058] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4757.685953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4758.779566] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4758.790323] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4758.826547] Lustre: lustre-MDT0000: new disk, initializing [ 4758.850697] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4758.857653] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4760.258714] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 4764.371170] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4764.416774] Lustre: 320329:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4764.465402] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4764.468043] Lustre: Skipped 1 previous similar message [ 4764.504765] Lustre: lustre-MDT0001: new disk, initializing [ 4764.538912] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4764.552449] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4764.555660] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4765.902257] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 4768.971107] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4770.149433] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4772.468380] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4772.567397] Lustre: lustre-OST0000: new disk, initializing [ 4772.569164] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4772.594974] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4774.390983] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 4777.542545] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4778.876686] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4781.051846] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4781.056799] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4781.089453] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4782.012593] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 4783.262429] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4783.331360] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4786.147519] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4786.149567] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4786.150668] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4786.155803] Lustre: Skipped 1 previous similar message [ 4790.032553] Lustre: server umount lustre-OST0000 complete [ 4795.360054] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4795.362317] LustreError: Skipped 1 previous similar message [ 4795.363539] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4795.367574] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4795.370336] Lustre: Skipped 1 previous similar message [ 4796.384686] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4796.385627] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4797.470616] Lustre: server umount lustre-MDT0000 complete [ 4798.673181] LustreError: 320320:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773106629 with bad export cookie 2672918922092116206 [ 4798.676325] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4798.676381] LustreError: 320320:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4798.790569] Lustre: server umount lustre-MDT0001 complete [ 4801.551474] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 21:37:12 (1773106632) [ 4804.077662] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 21:37:14 (1773106634) [ 4806.875879] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4807.029441] LustreError: 324694:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4807.035210] LustreError: 324694:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 4807.055757] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4808.376427] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 4811.213355] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4811.333047] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4812.634534] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 4814.417084] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4815.593183] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4817.858029] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4820.257134] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 4821.973616] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4841.166634] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 4841.728929] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 4842.283155] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 4842.858646] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 4843.594674] Lustre: Failing over lustre-MDT0000 [ 4843.799683] Lustre: server umount lustre-MDT0000 complete [ 4844.002391] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4844.004127] LustreError: 324719:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4844.006184] Lustre: Skipped 1 previous similar message [ 4847.989429] LustreError: 324691:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4847.996674] LustreError: 324691:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 4853.101349] LustreError: 324689:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4853.106053] LustreError: 324689:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 4856.887814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4856.931512] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4857.025269] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4857.027270] Lustre: Skipped 1 previous similar message [ 4857.054584] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4858.221719] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4858.285557] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 4862.441352] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4862.445535] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4862.465226] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 4863.642841] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4864.319124] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4866.208393] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 4866.929369] Lustre: Failing over lustre-MDT0000 [ 4867.091097] Lustre: server umount lustre-MDT0000 complete [ 4867.552945] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4867.555021] LustreError: 324690:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4867.559285] Lustre: Skipped 3 previous similar messages [ 4867.567757] LustreError: 324690:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 4 previous similar messages [ 4877.793104] LustreError: 324695:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4877.798961] LustreError: 324695:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 7 previous similar messages [ 4880.133899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4880.185054] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4880.286949] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4880.312079] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4881.515371] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 4883.823848] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4885.484668] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4885.487873] Lustre: Skipped 2 previous similar messages [ 4885.497840] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 4885.525408] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 4887.004694] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4887.880301] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4895.714012] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4895.715658] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4895.716531] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4895.716536] Lustre: Skipped 1 previous similar message [ 4895.720414] Lustre: Skipped 1 previous similar message [ 4896.809769] Lustre: server umount lustre-MDT0000 complete [ 4898.062214] LustreError: 325412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773106728 with bad export cookie 2672918922092118901 [ 4898.065568] LustreError: 325412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4898.066354] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4898.172337] Lustre: server umount lustre-MDT0001 complete [ 4909.824307] Lustre: server umount lustre-OST0000 complete [ 4916.165853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4916.341409] LustreError: 330491:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4916.347726] LustreError: 330491:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 4916.366607] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4917.655801] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 4920.806187] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4922.379740] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 4924.453936] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4925.694102] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4928.159270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4930.124639] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 4931.874266] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4940.594421] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 4945.892058] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4945.893915] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4945.894916] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4945.898918] Lustre: Skipped 2 previous similar messages [ 4945.901983] Lustre: Skipped 2 previous similar messages [ 4949.458479] Lustre: server umount lustre-OST0000 complete [ 4956.128358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4956.131023] Lustre: Skipped 4 previous similar messages [ 4965.343107] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4965.406590] Lustre: server umount lustre-MDT0000 complete [ 4966.368351] LustreError: 331201:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4966.375183] LustreError: 331201:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 4966.858310] LustreError: 330473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773106797 with bad export cookie 2672918922092119622 [ 4966.862669] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4966.864310] LustreError: 330473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4966.992668] Lustre: server umount lustre-MDT0001 complete [ 4970.755357] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 4971.728612] Key type lgssc unregistered [ 4971.856413] LNet: 333699:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4971.858830] LNetError: 333699:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4971.865281] LNet: Removed LNI 192.168.202.131@tcp [ 4972.155323] Key type .llcrypt unregistered [ 4972.156379] Key type ._llcrypt unregistered [ 4981.527255] Key type ._llcrypt registered [ 4981.528283] Key type .llcrypt registered [ 4981.568807] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 4981.923901] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4981.930938] alg: No test for adler32 (adler32-zlib) [ 4982.785201] Lustre: Lustre: Build Version: 2.17.50_236_gea5aa0c [ 4982.861615] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 4984.439134] Key type lgssc registered [ 4984.862915] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4987.469640] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4987.476156] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4988.629860] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4989.817385] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 4992.951330] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4993.065380] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4994.513318] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 4996.920332] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4998.228206] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5000.923210] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5001.060107] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5003.257513] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 5005.336670] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5014.374692] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 5023.712286] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5023.716561] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5023.723760] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5024.738275] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5024.741920] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5025.948984] Lustre: server umount lustre-MDT0000 complete [ 5027.125165] LustreError: 337436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773106857 with bad export cookie 11378957717238359053 [ 5027.128491] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5027.128948] LustreError: 337436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5027.236735] Lustre: server umount lustre-MDT0001 complete [ 5038.802982] Lustre: server umount lustre-OST0000 complete [ 5043.126944] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5043.279967] LustreError: 338568:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5043.285171] LustreError: 338568:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 5043.303942] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5044.514798] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 5047.747271] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5047.889097] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5049.142702] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 5051.102283] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5052.534552] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5054.953397] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5056.916266] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 5058.775492] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5068.389414] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5078.496050] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5078.498210] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5078.501246] Lustre: Skipped 1 previous similar message [ 5078.502807] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5078.504314] Lustre: Skipped 2 previous similar messages [ 5082.138967] Lustre: server umount lustre-MDT0000 complete [ 5083.338206] LustreError: 338548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773106913 with bad export cookie 11378957717238360152 [ 5083.340503] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5083.341316] LustreError: 338548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5083.468413] Lustre: server umount lustre-MDT0001 complete [ 5095.009108] Lustre: server umount lustre-OST0000 complete [ 5100.007900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5100.152695] LustreError: 342161:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5100.173311] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5100.175121] Lustre: Skipped 1 previous similar message [ 5101.375291] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 5104.102961] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5105.499047] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 5107.188568] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5108.281098] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5110.473096] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5111.591140] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5112.488384] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 5114.244288] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5116.896186] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5116.900223] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5116.906324] Lustre: Skipped 2 previous similar messages [ 5116.909268] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5116.911367] Lustre: Skipped 3 previous similar messages [ 5118.944707] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5121.481044] Lustre: server umount lustre-OST0000 complete [ 5124.064973] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5124.070149] Lustre: Skipped 1 previous similar message [ 5124.073079] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5124.076372] Lustre: Skipped 3 previous similar messages [ 5128.790867] Lustre: server umount lustre-MDT0000 complete [ 5129.183901] LustreError: 342161:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5129.190191] LustreError: 342161:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 5129.968809] LustreError: 342186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773106960 with bad export cookie 11378957717238361160 [ 5129.972020] LustreError: 342186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5129.972403] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5130.089273] Lustre: server umount lustre-MDT0001 complete [ 5134.026883] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 5134.991874] Key type lgssc unregistered [ 5135.118368] LNet: 345335:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5135.120850] LNetError: 345335:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5135.128293] LNet: Removed LNI 192.168.202.131@tcp [ 5135.414995] Key type .llcrypt unregistered [ 5135.416141] Key type ._llcrypt unregistered [ 5144.867536] Key type ._llcrypt registered [ 5144.868567] Key type .llcrypt registered [ 5144.908832] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 5145.424854] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5145.432129] alg: No test for adler32 (adler32-zlib) [ 5146.273494] Lustre: Lustre: Build Version: 2.17.50_236_gea5aa0c [ 5146.350326] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 5147.927150] Key type lgssc registered [ 5148.281892] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5150.722563] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5150.728381] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5150.731968] LustreError: 346735:0:(osd_handler.c:8547:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 5150.740811] LustreError: 346735:0:(obd_config.c:880:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 5150.743667] LustreError: 346735:0:(obd_mount.c:217:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 5150.746563] LustreError: 346735:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 5150.750099] LustreError: 346735:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5153.225828] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5153.228137] LustreError: 347131:0:(osd_handler.c:8547:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 5153.235856] LustreError: 347131:0:(obd_config.c:880:class_setup()) setup lustre-OST0000-osd failed (-22) [ 5153.237870] LustreError: 347131:0:(obd_mount.c:217:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 5153.240070] LustreError: 347131:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 5153.242196] LustreError: 347131:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5159.499647] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 5160.450738] Key type lgssc unregistered [ 5160.582346] LNet: 348115:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5160.584818] LNetError: 348115:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5160.594260] LNet: Removed LNI 192.168.202.131@tcp [ 5160.897317] Key type .llcrypt unregistered [ 5160.898699] Key type ._llcrypt unregistered [ 5166.323083] Key type ._llcrypt registered [ 5166.324145] Key type .llcrypt registered [ 5166.362442] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_hostid [ 5172.254311] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 5172.563276] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5172.569957] alg: No test for adler32 (adler32-zlib) [ 5173.421673] Lustre: Lustre: Build Version: 2.17.50_236_gea5aa0c [ 5173.505621] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 5175.087126] Key type lgssc registered [ 5175.428643] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5185.919857] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5185.924719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5187.011281] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5187.021442] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5187.058108] Lustre: lustre-MDT0000: new disk, initializing [ 5187.081983] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5187.087997] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5188.309524] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 5192.276633] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5192.319195] Lustre: 351209:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5192.347310] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5192.349130] Lustre: Skipped 1 previous similar message [ 5192.389552] Lustre: lustre-MDT0001: new disk, initializing [ 5192.422371] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5192.434897] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5192.438833] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5193.662410] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 5196.548750] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5197.645186] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5199.738706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5199.834848] Lustre: lustre-OST0000: new disk, initializing [ 5199.836620] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5199.863853] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5201.559104] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 5203.194211] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5203.197041] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5203.231240] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5204.481752] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5205.711433] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5205.773581] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5206.918243] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5206.978714] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5208.544616] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5208.545551] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5208.546724] LustreError: Skipped 1 previous similar message [ 5208.551290] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5208.553273] Lustre: Skipped 1 previous similar message [ 5209.614227] Lustre: server umount lustre-OST0000 complete [ 5213.151829] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5213.153794] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5213.157084] Lustre: Skipped 1 previous similar message [ 5213.158576] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5216.919053] Lustre: server umount lustre-MDT0000 complete [ 5218.053157] LustreError: 351202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773107048 with bad export cookie 3795201987747839715 [ 5218.055609] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5218.059154] LustreError: 351202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5218.173930] Lustre: server umount lustre-MDT0001 complete [ 5220.843333] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 5221.389626] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 21:44:12 (1773107052) [ 5223.442285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5223.613247] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5224.851260] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 5227.679620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5227.791056] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5228.997929] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 5230.618447] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5231.678779] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5233.825558] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5235.816345] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 5237.486452] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5239.616818] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5239.679285] Lustre: lustre-OST0001: new disk, initializing [ 5239.682528] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5239.712748] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5239.715170] Lustre: Skipped 1 previous similar message [ 5241.427064] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 5244.345348] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5244.424769] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5244.428249] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5244.465103] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5249.509853] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5249.513460] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5249.514738] LustreError: Skipped 1 previous similar message [ 5249.518093] Lustre: Skipped 1 previous similar message [ 5249.518751] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5249.524008] Lustre: Skipped 2 previous similar messages [ 5252.223247] Lustre: server umount lustre-OST0001 complete [ 5254.627240] LustreError: 357469:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5254.631862] LustreError: 357469:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 5259.747831] LustreError: 358564:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5259.751930] LustreError: 358564:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 5264.865783] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5264.866321] LustreError: 356849:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5264.867828] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5264.867836] Lustre: Skipped 1 previous similar message [ 5264.869550] Lustre: Skipped 2 previous similar messages [ 5264.875250] LustreError: 356849:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 5269.986493] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5269.986831] LustreError: 357469:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5269.988215] Lustre: Skipped 2 previous similar messages [ 5269.995456] LustreError: 357469:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 5270.353190] Lustre: server umount lustre-OST0000 complete [ 5274.079898] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5274.082179] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5277.787167] Lustre: server umount lustre-MDT0000 complete [ 5279.028306] LustreError: 355239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773107109 with bad export cookie 3795201987747840583 [ 5279.031773] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5279.032774] LustreError: 355239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5285.230117] Lustre: server umount lustre-MDT0001 complete [ 5288.563220] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 5289.599374] Key type lgssc unregistered [ 5289.722294] LNet: 359606:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5289.725433] LNetError: 359606:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5289.735313] LNet: Removed LNI 192.168.202.131@tcp [ 5290.006336] Key type .llcrypt unregistered [ 5290.007434] Key type ._llcrypt unregistered [ 5308.564746] Key type ._llcrypt registered [ 5308.565745] Key type .llcrypt registered [ 5308.611585] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 5309.174435] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5309.187674] alg: No test for adler32 (adler32-zlib) [ 5310.034949] Lustre: Lustre: Build Version: 2.17.50_236_gea5aa0c [ 5310.112323] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 5311.687215] Key type lgssc registered [ 5312.154420] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5314.728664] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5314.734320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5315.823713] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5315.831778] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5315.889496] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5317.126135] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 5320.277247] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5320.310722] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5320.322202] Lustre: 362365:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5320.416521] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5321.704362] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 5323.944716] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5325.236649] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5328.866425] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5328.867341] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5328.874629] Lustre: Skipped 1 previous similar message [ 5328.880702] Lustre: Skipped 1 previous similar message [ 5332.323116] Lustre: server umount lustre-MDT0000 complete [ 5333.836506] LustreError: 362357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773107164 with bad export cookie 16773178554175666844 [ 5333.842532] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5333.974123] Lustre: server umount lustre-MDT0001 complete [ 5337.126519] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 21:46:08 (1773107168) [ 5339.240086] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5339.410982] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5340.644102] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 5343.489939] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5343.675993] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5345.392096] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 5347.486571] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5348.716233] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5351.576406] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5351.688485] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5351.801729] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5353.783532] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 5355.476143] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5359.276009] Lustre: Permanently deactivating lustre-OST0000 [ 5359.281079] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 5360.817716] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5361.242799] LustreError: 367146:0:(mgs_llog.c:1942:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 5361.246249] LustreError: 367146:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [ 5361.248397] LustreError: 367146:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5361.860220] LustreError: 367194:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [ 5361.862811] LustreError: 367194:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5368.592829] Lustre: server umount lustre-OST0000 complete [ 5369.498437] LustreError: 367445:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [ 5369.502490] LustreError: 367445:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5372.384527] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5372.385198] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5372.389180] Lustre: Skipped 1 previous similar message [ 5372.392653] Lustre: Skipped 1 previous similar message [ 5377.504566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5377.507847] Lustre: Skipped 1 previous similar message [ 5379.590543] Lustre: server umount lustre-MDT0000 complete [ 5380.816039] LustreError: 364672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773107211 with bad export cookie 16773178554175667411 [ 5380.818460] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5380.819491] LustreError: 364672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 5391.375150] Lustre: server umount lustre-MDT0001 complete [ 5394.200726] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5395.483827] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 5396.339576] LustreError: 368584:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 5398.393821] Lustre: 368922:0:(mgs_llog.c:1709:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.131@tcp [ 5399.026454] Lustre: server umount MGS complete [ 5402.652392] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5402.796886] LustreError: 369645:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5402.801597] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5402.801687] LustreError: 369645:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 5402.804876] Lustre: Skipped 1 previous similar message [ 5402.826991] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5404.054748] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 5406.987925] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5407.102560] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5408.555485] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 5410.335472] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5411.686735] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5412.393459] Lustre: Permanently reactivating lustre-OST0000 [ 5412.400303] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 5412.402339] Lustre: Skipped 2 previous similar messages [ 5435.871202] LustreError: 371152:0:(osp_dev.c:694:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 5438.678499] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5438.819317] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5438.822923] Lustre: Skipped 1 previous similar message [ 5440.853826] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 5442.809491] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5456.355307] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5456.355358] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5456.356196] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5456.358218] LustreError: Skipped 1 previous similar message [ 5456.364991] Lustre: Skipped 1 previous similar message [ 5460.571062] Lustre: server umount lustre-OST0000 complete [ 5463.519874] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5463.522396] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5463.526302] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5463.527913] Lustre: Skipped 1 previous similar message [ 5471.712041] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5471.713725] Lustre: Skipped 3 previous similar messages [ 5476.319092] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5476.398463] Lustre: server umount lustre-MDT0000 complete [ 5476.832173] LustreError: 369645:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5476.835989] LustreError: 369645:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 5477.789723] LustreError: 369625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773107308 with bad export cookie 16773178554175668433 [ 5477.793551] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5477.795576] LustreError: 369625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5477.927431] Lustre: server umount lustre-MDT0001 complete [ 5481.538215] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 5482.534192] Key type lgssc unregistered [ 5482.667610] LNet: 373868:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5482.671977] LNetError: 373868:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5482.683401] LNet: Removed LNI 192.168.202.131@tcp [ 5483.041845] Key type .llcrypt unregistered [ 5483.043900] Key type ._llcrypt unregistered [ 5489.128968] Key type ._llcrypt registered [ 5489.129927] Key type .llcrypt registered [ 5489.166781] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_hostid [ 5495.498689] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 5495.875177] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5495.906674] alg: No test for adler32 (adler32-zlib) [ 5496.756589] Lustre: Lustre: Build Version: 2.17.50_236_gea5aa0c [ 5496.842060] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 5498.423165] Key type lgssc registered [ 5498.752491] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5509.399398] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5509.403794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5510.495695] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5510.508211] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5510.545832] Lustre: lustre-MDT0000: new disk, initializing [ 5510.569574] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5510.576150] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5511.756865] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 5515.664199] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5515.707816] Lustre: 376964:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5515.729637] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5515.731359] Lustre: Skipped 1 previous similar message [ 5515.767902] Lustre: lustre-MDT0001: new disk, initializing [ 5515.798809] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5515.813780] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5515.816852] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5516.996994] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 5520.157828] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5521.207198] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5523.638589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5523.739856] Lustre: lustre-OST0000: new disk, initializing [ 5523.741707] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5523.765952] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5525.301966] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5525.304821] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5525.342282] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5525.612170] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 5528.543535] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5529.804321] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5529.866810] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5531.087553] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5531.153157] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5533.900197] Lustre: server umount lustre-OST0000 complete [ 5535.712734] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5535.714114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5535.719770] Lustre: Skipped 1 previous similar message [ 5540.832450] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5540.834517] Lustre: Skipped 2 previous similar messages [ 5544.955886] Lustre: server umount lustre-MDT0000 complete [ 5546.163738] LustreError: 378702:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773107376 with bad export cookie 15125285404951438992 [ 5546.167299] LustreError: 378702:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5546.167820] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5555.746274] Lustre: server umount lustre-MDT0001 complete [ 5558.838392] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 5559.443202] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 21:49:50 (1773107390) [ 5561.656166] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5561.828454] LustreError: 381010:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5561.835153] LustreError: 381010:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 5561.858642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5563.200569] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 5565.925336] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5566.089916] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5567.378367] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 5569.483670] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5570.624629] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5572.789638] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5574.702294] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 5576.449385] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5577.113814] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 5593.024438] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 5596.639891] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5596.642540] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5596.648450] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5598.688583] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5598.690686] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5598.690838] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5598.695377] Lustre: Skipped 1 previous similar message [ 5600.659537] Lustre: server umount lustre-OST0000 complete [ 5602.272280] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5602.274243] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5603.808614] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5603.810495] Lustre: Skipped 2 previous similar messages [ 5608.090318] Lustre: server umount lustre-MDT0000 complete [ 5608.927954] LustreError: 381011:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5608.933641] LustreError: 381011:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 5609.323303] LustreError: 382624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773107439 with bad export cookie 15125285404951439860 [ 5609.327021] LustreError: 382624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5609.327234] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5609.454258] Lustre: server umount lustre-MDT0001 complete [ 5613.137742] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 5614.188777] Key type lgssc unregistered [ 5614.328409] LNet: 384401:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5614.331902] LNetError: 384401:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5614.341319] LNet: Removed LNI 192.168.202.131@tcp [ 5614.615761] Key type .llcrypt unregistered [ 5614.617301] Key type ._llcrypt unregistered [ 5621.833378] Key type ._llcrypt registered [ 5621.834424] Key type .llcrypt registered [ 5621.878446] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 5621.978348] Key type .llcrypt unregistered [ 5621.979986] Key type ._llcrypt unregistered [ 5628.703280] Key type ._llcrypt registered [ 5628.704713] Key type .llcrypt registered [ 5628.740399] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 5629.106715] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5629.139838] alg: No test for adler32 (adler32-zlib) [ 5629.982863] Lustre: Lustre: Build Version: 2.17.50_236_gea5aa0c [ 5630.059099] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 5631.639134] Key type lgssc registered [ 5631.969654] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5634.339970] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5634.345795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5635.492205] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5636.617842] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 5639.143695] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5639.249130] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5641.110047] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 5642.764953] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5644.256320] LustreError: 386555:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5644.260969] LustreError: 386555:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 4 previous similar messages [ 5644.866616] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5644.984678] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5645.991832] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5646.199795] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 5650.400168] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5650.402286] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5650.405843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5651.425520] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5651.427390] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5651.427404] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5655.185694] Lustre: server umount lustre-OST0000 complete [ 5656.544289] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5656.544855] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5656.547419] Lustre: Skipped 1 previous similar message [ 5656.550038] Lustre: Skipped 1 previous similar message [ 5661.664635] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5661.668237] Lustre: Skipped 1 previous similar message [ 5662.517863] Lustre: server umount lustre-MDT0000 complete [ 5663.795411] LustreError: 386534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773107494 with bad export cookie 6049065590601951189 [ 5663.799335] LustreError: 386534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5663.799628] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5663.918801] Lustre: server umount lustre-MDT0001 complete [ 5667.441980] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 5668.459943] Key type lgssc unregistered [ 5668.578294] LNet: 389541:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5668.580475] LNetError: 389541:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5668.588298] LNet: Removed LNI 192.168.202.131@tcp [ 5668.855324] Key type .llcrypt unregistered [ 5668.856779] Key type ._llcrypt unregistered [ 5677.305982] Key type ._llcrypt registered [ 5677.307032] Key type .llcrypt registered [ 5677.340979] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 5677.599408] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5677.613833] alg: No test for adler32 (adler32-zlib) [ 5678.462024] Lustre: Lustre: Build Version: 2.17.50_236_gea5aa0c [ 5678.539296] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 5680.111225] Key type lgssc registered [ 5680.458045] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5682.889760] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5682.896207] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5698.335445] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5703.455222] LustreError: 390891:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.131@tcp: failed processing log, type 4: rc = -110 [ 5730.079198] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5732.197365] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 5733.986567] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5736.327867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5736.509771] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5736.531771] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 5737.788409] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 5740.887314] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5741.012704] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5742.374690] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 5744.168343] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5745.262551] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5750.242550] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5750.243707] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5750.248270] Lustre: Skipped 1 previous similar message [ 5750.253510] Lustre: Skipped 1 previous similar message [ 5753.548209] Lustre: server umount lustre-OST0000 complete [ 5755.361499] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5755.362203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5755.366618] Lustre: Skipped 1 previous similar message [ 5760.480493] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5760.482312] Lustre: Skipped 2 previous similar messages [ 5760.920746] Lustre: server umount lustre-MDT0000 complete [ 5762.143389] LustreError: 390898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773107592 with bad export cookie 3953610940582577514 [ 5762.147538] LustreError: 390898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5762.148345] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5762.269881] Lustre: server umount lustre-MDT0001 complete [ 5765.554199] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 5766.518044] Key type lgssc unregistered [ 5766.637366] LNet: 394105:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5766.640789] LNetError: 394105:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5766.648325] LNet: Removed LNI 192.168.202.131@tcp [ 5766.933305] Key type .llcrypt unregistered [ 5766.934296] Key type ._llcrypt unregistered [ 5775.877555] Key type ._llcrypt registered [ 5775.878509] Key type .llcrypt registered [ 5775.916992] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 5776.401793] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5776.435282] alg: No test for adler32 (adler32-zlib) [ 5777.296328] Lustre: Lustre: Build Version: 2.17.50_236_gea5aa0c [ 5777.386149] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 5778.975159] Key type lgssc registered [ 5779.339146] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5781.827255] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5781.833550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5782.980749] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5784.169109] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 5786.786906] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5786.921201] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5788.094709] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 5789.693055] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5790.774422] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5792.905527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5793.015734] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5794.021824] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 5794.731896] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 5796.460245] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5802.465693] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5802.465987] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5802.467321] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5802.470066] Lustre: Skipped 1 previous similar message [ 5804.511184] Lustre: server umount lustre-MDT0000 complete [ 5805.273277] LustreError: 395462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.31@tcp arrived at 1773107635 with bad export cookie 10287544165992297549 [ 5805.277434] LustreError: 395462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5805.916912] LustreError: 397093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773107636 with bad export cookie 10287544165992296702 [ 5805.921689] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5807.584949] LustreError: 396200:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5807.585326] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5807.585678] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5807.585688] Lustre: Skipped 1 previous similar message [ 5807.586462] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5807.586466] Lustre: Skipped 3 previous similar messages [ 5807.588953] LustreError: 396200:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 5810.591986] LustreError: 395480:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5810.592303] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5810.597181] LustreError: 395480:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 5812.051963] Lustre: server umount lustre-OST0000 complete [ 5814.394955] Lustre: server umount lustre-MDT0001 complete [ 5818.208588] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 5819.285757] Key type lgssc unregistered [ 5819.413436] LNet: 398808:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5819.416071] LNetError: 398808:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5819.424338] LNet: Removed LNI 192.168.202.131@tcp [ 5819.738776] Key type .llcrypt unregistered [ 5819.740113] Key type ._llcrypt unregistered [ 5828.446148] Key type ._llcrypt registered [ 5828.447359] Key type .llcrypt registered [ 5828.480447] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 5828.805741] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5828.849438] alg: No test for adler32 (adler32-zlib) [ 5829.700755] Lustre: Lustre: Build Version: 2.17.50_236_gea5aa0c [ 5829.784504] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 5831.375267] Key type lgssc registered [ 5831.766052] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5834.324579] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5834.330338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5835.499585] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5836.739467] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 5839.420125] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5839.534250] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5840.742770] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 5842.385875] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5843.489766] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5845.535435] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5845.641870] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5846.695641] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 5847.348299] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 5848.990065] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5855.202099] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 5855.202291] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5855.204968] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5855.208482] Lustre: Skipped 1 previous similar message [ 5855.211256] Lustre: Skipped 1 previous similar message [ 5856.912275] Lustre: server umount lustre-MDT0001 complete [ 5860.320248] LustreError: 400182:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5860.321270] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5860.326097] LustreError: 400182:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 5860.329745] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5860.333221] Lustre: Skipped 1 previous similar message [ 5864.622236] Lustre: server umount lustre-OST0000 complete [ 5865.441802] LustreError: 400183:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5865.963502] Lustre: server umount lustre-MDT0000 complete [ 5870.610892] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 5871.638299] Key type lgssc unregistered [ 5871.782382] LNet: 403509:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5871.784832] LNetError: 403509:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5871.794284] LNet: Removed LNI 192.168.202.131@tcp [ 5872.103381] Key type .llcrypt unregistered [ 5872.104411] Key type ._llcrypt unregistered [ 5879.812151] Key type ._llcrypt registered [ 5879.813112] Key type .llcrypt registered [ 5879.847286] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_hostid [ 5885.562702] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 5885.964905] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5885.979730] alg: No test for adler32 (adler32-zlib) [ 5886.823539] Lustre: Lustre: Build Version: 2.17.50_236_gea5aa0c [ 5886.902892] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 5888.479129] Key type lgssc registered [ 5888.833575] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5900.735162] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 5905.665573] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5905.670960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5906.764829] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5906.776477] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5906.814034] Lustre: lustre-MDT0000: new disk, initializing [ 5906.838062] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5906.845317] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5908.102574] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 5913.927851] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5914.044584] Lustre: lustre-OST0000: new disk, initializing [ 5914.046471] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5914.048079] Lustre: Skipped 1 previous similar message [ 5914.073951] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5915.947428] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 5919.017833] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5919.743362] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 5919.747547] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 5919.755369] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 5929.407214] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5929.449789] Lustre: 407443:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5929.477417] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5929.517936] Lustre: lustre-MDT0001: new disk, initializing [ 5929.557106] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5929.569736] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 5929.572766] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 5930.830680] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 5934.411588] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 5934.490636] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 5935.712393] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 5936.796988] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 1 sec [ 5941.217523] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5941.218054] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 5941.221533] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5941.224599] Lustre: Skipped 1 previous similar message [ 5942.132125] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.31@tcp (stopping) [ 5944.400761] Lustre: server umount lustre-MDT0001 complete [ 5951.458285] LustreError: 407450:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5951.463466] LustreError: 407450:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 4 previous similar messages [ 5956.577956] LustreError: 408452:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5956.589199] LustreError: 408452:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 5960.672043] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5960.676396] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5960.682276] Lustre: Skipped 1 previous similar message [ 5960.685179] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5961.697522] LustreError: 407455:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5962.842195] Lustre: server umount lustre-OST0000 complete [ 5970.415974] Lustre: server umount lustre-MDT0000 complete [ 5974.739856] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 5975.811187] Key type lgssc unregistered [ 5975.930379] LNet: 412020:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5975.933860] LNetError: 412020:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5975.944304] LNet: Removed LNI 192.168.202.131@tcp [ 5976.233348] Key type .llcrypt unregistered [ 5976.234408] Key type ._llcrypt unregistered [ 5988.632295] Key type ._llcrypt registered [ 5988.633283] Key type .llcrypt registered [ 5988.668312] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 5988.754396] Key type .llcrypt unregistered [ 5988.755401] Key type ._llcrypt unregistered [ 5995.485190] Key type ._llcrypt registered [ 5995.486485] Key type .llcrypt registered [ 5995.523398] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 5995.847485] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5995.899459] alg: No test for adler32 (adler32-zlib) [ 5996.762523] Lustre: Lustre: Build Version: 2.17.50_236_gea5aa0c [ 5996.837182] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 5998.415197] Key type lgssc registered [ 5998.995328] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6007.772270] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6007.776770] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6008.865286] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6008.874872] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6008.911438] Lustre: lustre-MDT0000: new disk, initializing [ 6008.933501] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6008.939398] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6010.144835] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 6014.283149] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6014.321383] Lustre: 415333:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6014.352320] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6014.356149] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6014.357633] Lustre: Skipped 1 previous similar message [ 6014.393139] Lustre: lustre-MDT0001: new disk, initializing [ 6014.424357] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6014.435587] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6014.438821] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6015.560035] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 6018.441410] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6019.677643] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6022.168669] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6022.310213] Lustre: lustre-OST0000: new disk, initializing [ 6022.312529] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6022.341987] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6022.909853] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6022.916291] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6022.957854] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6024.270588] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 6027.511966] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6029.791758] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6029.794088] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6029.798292] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6033.377435] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6033.378271] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6033.378279] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6033.379869] Lustre: Skipped 1 previous similar message [ 6035.891980] Lustre: server umount lustre-MDT0000 complete [ 6037.154045] LustreError: 416113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773107867 with bad export cookie 16659416933001535514 [ 6037.157683] LustreError: 416113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6037.157746] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6037.264596] Lustre: server umount lustre-MDT0001 complete [ 6048.672342] Lustre: server umount lustre-OST0000 complete [ 6052.467373] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 21:58:03 (1773107883) [ 6054.967511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6055.149087] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6056.494765] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 6059.135670] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6059.251556] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6060.558778] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 6062.959717] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6064.505819] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6080.673678] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6080.743073] Lustre: Found index 0 for lustre-OST0000, updating log [ 6080.803356] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6081.829774] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 6082.549722] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 6084.195204] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6087.135766] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6087.138153] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6087.142038] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6089.184851] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6091.463753] Lustre: server umount lustre-OST0000 complete [ 6103.894071] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6103.894695] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6103.897736] Lustre: Skipped 2 previous similar messages [ 6103.899347] Lustre: Skipped 2 previous similar messages [ 6116.320764] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6116.324324] Lustre: Skipped 8 previous similar messages [ 6118.367166] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6118.454115] Lustre: server umount lustre-MDT0000 complete [ 6119.665837] LustreError: 418970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773107950 with bad export cookie 16659416933001536879 [ 6119.671489] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6119.672359] LustreError: 418970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6119.799476] Lustre: server umount lustre-MDT0001 complete [ 6123.052484] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 21:59:13 (1773107953) [ 6145.311194] LNet: 1 local NIs in recovery (showing 1): 192.168.202.131@tcp [ 6702.190990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6702.344586] LustreError: 425067:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6702.350490] LustreError: 425067:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 6702.369055] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6703.610361] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 6706.210702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6707.620740] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 6709.308687] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6710.376420] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6725.538759] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6725.635808] Lustre: Found index 0 for lustre-OST0000, updating log [ 6727.593502] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 6728.869861] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [ 6729.361055] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6732.067683] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6732.071535] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6736.141881] Lustre: server umount lustre-OST0000 complete [ 6737.375869] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6737.378029] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6743.452152] Lustre: server umount lustre-MDT0000 complete [ 6743.522135] LustreError: 425086:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6743.530246] LustreError: 425086:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 6744.545494] LustreError: 425067:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6744.549547] LustreError: 425067:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 6744.643628] LustreError: 425050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773108575 with bad export cookie 16659416933001537950 [ 6744.648142] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6744.649495] LustreError: 425050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6744.784498] Lustre: server umount lustre-MDT0001 complete [ 6759.712180] LNet: 1 local NIs in recovery (showing 1): 192.168.202.131@tcp [ 6761.263332] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_hostid [ 6764.008613] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 6774.411097] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 22:10:05 (1773108605) [ 6779.623561] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 6780.531238] Key type lgssc unregistered [ 6787.807663] LNet: 431332:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6787.810031] LNetError: 431332:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6858.469295] LNet: Removed LNI 192.168.202.131@tcp [ 6858.732178] Key type .llcrypt unregistered [ 6858.733217] Key type ._llcrypt unregistered [ 6865.214413] Key type ._llcrypt registered [ 6865.215243] Key type .llcrypt registered [ 6865.253980] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 6865.739883] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6865.753693] alg: No test for adler32 (adler32-zlib) [ 6866.599503] Lustre: Lustre: Build Version: 2.17.50_236_gea5aa0c [ 6866.676582] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 6868.255187] Key type lgssc registered [ 6868.805362] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6870.569478] LNet: Added LNI 192.168.202.131@tcp100 [8/256/0/180] [ 6870.933961] LNet: Added LNI 192.168.202.131@tcp101 [8/256/0/180] [ 6871.310937] LNet: Added LNI 192.168.202.131@tcp102 [8/256/0/180] [ 6871.688481] LNet: Added LNI 192.168.202.131@tcp103 [8/256/0/180] [ 6872.102776] LNet: Added LNI 192.168.202.131@tcp104 [8/256/0/180] [ 6872.555302] LNet: Added LNI 192.168.202.131@tcp105 [8/256/0/180] [ 6872.986675] LNet: Added LNI 192.168.202.131@tcp106 [8/256/0/180] [ 6873.411913] LNet: Added LNI 192.168.202.131@tcp107 [8/256/0/180] [ 6873.850754] LNet: Added LNI 192.168.202.131@tcp108 [8/256/0/180] [ 6874.225797] LNet: Added LNI 192.168.202.131@tcp109 [8/256/0/180] [ 6874.578934] LNet: Added LNI 192.168.202.131@tcp110 [8/256/0/180] [ 6874.928784] LNet: Added LNI 192.168.202.131@tcp111 [8/256/0/180] [ 6875.326441] LNet: Added LNI 192.168.202.131@tcp112 [8/256/0/180] [ 6875.766473] LNet: Added LNI 192.168.202.131@tcp113 [8/256/0/180] [ 6876.217574] LNet: Added LNI 192.168.202.131@tcp114 [8/256/0/180] [ 6876.642594] LNet: Added LNI 192.168.202.131@tcp115 [8/256/0/180] [ 6877.078867] LNet: Added LNI 192.168.202.131@tcp116 [8/256/0/180] [ 6877.517691] LNet: Added LNI 192.168.202.131@tcp117 [8/256/0/180] [ 6877.926631] LNet: Added LNI 192.168.202.131@tcp118 [8/256/0/180] [ 6878.234263] LNet: Added LNI 192.168.202.131@tcp119 [8/256/0/180] [ 6878.548237] LNet: Added LNI 192.168.202.131@tcp120 [8/256/0/180] [ 6878.881388] LNet: Added LNI 192.168.202.131@tcp121 [8/256/0/180] [ 6879.199595] LNet: Added LNI 192.168.202.131@tcp122 [8/256/0/180] [ 6879.508896] LNet: Added LNI 192.168.202.131@tcp123 [8/256/0/180] [ 6879.922995] LNet: Added LNI 192.168.202.131@tcp124 [8/256/0/180] [ 6880.332893] LNet: Added LNI 192.168.202.131@tcp125 [8/256/0/180] [ 6880.773241] LNet: Added LNI 192.168.202.131@tcp126 [8/256/0/180] [ 6881.216829] LNet: Added LNI 192.168.202.131@tcp127 [8/256/0/180] [ 6881.627988] LNet: Added LNI 192.168.202.131@tcp128 [8/256/0/180] [ 6882.067661] LNet: Added LNI 192.168.202.131@tcp129 [8/256/0/180] [ 6882.518683] LNet: Added LNI 192.168.202.131@tcp130 [8/256/0/180] [ 6882.838820] LNet: Added LNI 192.168.202.131@tcp131 [8/256/0/180] [ 6883.114568] LNet: Added LNI 192.168.202.131@tcp132 [8/256/0/180] [ 6883.456520] LNet: Added LNI 192.168.202.131@tcp133 [8/256/0/180] [ 6883.762073] LNet: Added LNI 192.168.202.131@tcp134 [8/256/0/180] [ 6884.122341] LNet: Added LNI 192.168.202.131@tcp135 [8/256/0/180] [ 6887.020439] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6887.025675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6888.076453] Lustre: 434314:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.131@tcp: can't add NID 192.168.202.131@tcp130: rc = -75 [ 6888.141433] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6888.152207] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6888.197212] Lustre: lustre-MDT0000: new disk, initializing [ 6888.229126] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6888.237275] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6889.612338] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 6894.176950] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6894.263144] Lustre: 434330:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6894.308547] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6894.312703] Lustre: Skipped 1 previous similar message [ 6894.367389] Lustre: lustre-MDT0001: new disk, initializing [ 6894.381013] Lustre: 435115:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.131@tcp: can't add NID 192.168.202.131@tcp130: rc = -75 [ 6894.385359] Lustre: 435115:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 6894.432671] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6894.443922] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6894.447505] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6895.540618] Lustre: 435242:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.131@tcp: can't add NID 192.168.202.131@tcp130: rc = -75 [ 6895.544427] Lustre: 435242:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 6896.103770] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 6899.153369] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6900.203422] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6902.260547] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6902.373805] Lustre: lustre-OST0000: new disk, initializing [ 6902.375663] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6902.411091] Lustre: 436083:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.131@tcp: can't add NID 192.168.202.131@tcp130: rc = -75 [ 6902.413826] Lustre: 436083:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [ 6902.429760] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6904.148646] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 6905.211181] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6905.214027] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6905.253292] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6907.050201] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6910.433492] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6910.434759] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6910.437420] LustreError: Skipped 1 previous similar message [ 6910.442346] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6910.445686] Lustre: Skipped 1 previous similar message [ 6915.376688] Lustre: server umount lustre-OST0000 complete [ 6920.160523] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6920.165942] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6920.173797] Lustre: Skipped 1 previous similar message [ 6920.177368] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6922.974485] Lustre: server umount lustre-MDT0000 complete [ 6924.192515] LustreError: 434323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773108754 with bad export cookie 8396815257994894933 [ 6924.195751] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6924.196190] LustreError: 434323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6924.314647] Lustre: server umount lustre-MDT0001 complete [ 6927.858316] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 6928.902853] Key type lgssc unregistered [ 6929.060625] LNet: 437861:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6929.064309] LNetError: 437861:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6929.067457] LNet: Removed LNI 192.168.202.131@tcp [ 6930.079145] LNet: Removed LNI 192.168.202.131@tcp100 [ 6930.081562] LNetError: 437861:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6930.085662] LNetError: 437861:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 6930.088992] LNet: Removed LNI 192.168.202.131@tcp101 [ 6931.103139] LNet: Removed LNI 192.168.202.131@tcp102 [ 6931.105525] LNetError: 437861:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6931.109574] LNetError: 437861:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 6931.114033] LNet: Removed LNI 192.168.202.131@tcp103 [ 6932.127098] LNet: Removed LNI 192.168.202.131@tcp104 [ 6933.151093] LNet: Removed LNI 192.168.202.131@tcp105 [ 6933.153044] LNetError: 437861:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6933.156253] LNetError: 437861:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 6933.159761] LNet: Removed LNI 192.168.202.131@tcp106 [ 6934.175095] LNet: Removed LNI 192.168.202.131@tcp107 [ 6935.199082] LNet: Removed LNI 192.168.202.131@tcp108 [ 6936.223117] LNet: Removed LNI 192.168.202.131@tcp109 [ 6937.247110] LNet: Removed LNI 192.168.202.131@tcp110 [ 6937.248971] LNetError: 437861:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6937.252058] LNetError: 437861:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 6937.255497] LNet: Removed LNI 192.168.202.131@tcp111 [ 6937.257252] LNet: Removed LNI 192.168.202.131@tcp112 [ 6937.259039] LNet: Removed LNI 192.168.202.131@tcp113 [ 6937.260803] LNet: Removed LNI 192.168.202.131@tcp114 [ 6937.262516] LNet: Removed LNI 192.168.202.131@tcp115 [ 6937.264207] LNet: Removed LNI 192.168.202.131@tcp116 [ 6937.265356] LNet: Removed LNI 192.168.202.131@tcp117 [ 6937.266476] LNet: Removed LNI 192.168.202.131@tcp118 [ 6937.267559] LNet: Removed LNI 192.168.202.131@tcp119 [ 6937.268599] LNet: Removed LNI 192.168.202.131@tcp120 [ 6937.269616] LNet: Removed LNI 192.168.202.131@tcp121 [ 6937.270670] LNet: Removed LNI 192.168.202.131@tcp122 [ 6937.271768] LNet: Removed LNI 192.168.202.131@tcp123 [ 6937.272897] LNet: Removed LNI 192.168.202.131@tcp124 [ 6937.273960] LNet: Removed LNI 192.168.202.131@tcp125 [ 6937.275100] LNet: Removed LNI 192.168.202.131@tcp126 [ 6937.276135] LNet: Removed LNI 192.168.202.131@tcp127 [ 6937.277209] LNet: Removed LNI 192.168.202.131@tcp128 [ 6937.278243] LNet: Removed LNI 192.168.202.131@tcp129 [ 6937.279329] LNet: Removed LNI 192.168.202.131@tcp130 [ 6937.280328] LNet: Removed LNI 192.168.202.131@tcp131 [ 6937.281467] LNet: Removed LNI 192.168.202.131@tcp132 [ 6937.282494] LNet: Removed LNI 192.168.202.131@tcp133 [ 6937.283523] LNet: Removed LNI 192.168.202.131@tcp134 [ 6937.290363] LNet: Removed LNI 192.168.202.131@tcp135 [ 6937.531653] Key type .llcrypt unregistered [ 6937.532698] Key type ._llcrypt unregistered [ 6944.337441] Key type ._llcrypt registered [ 6944.338362] Key type .llcrypt registered [ 6944.370022] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_hostid [ 6949.910305] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 6950.202515] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6950.233056] alg: No test for adler32 (adler32-zlib) [ 6951.085847] Lustre: Lustre: Build Version: 2.17.50_236_gea5aa0c [ 6951.163718] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 6952.735120] Key type lgssc registered [ 6953.057618] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6963.071282] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 22:13:14 (1773108794) [ 6968.297195] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 6969.158797] Key type lgssc unregistered [ 6969.264329] LNet: 441865:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6969.266417] LNetError: 441865:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6969.275256] LNet: Removed LNI 192.168.202.131@tcp [ 6969.522388] Key type .llcrypt unregistered [ 6969.523981] Key type ._llcrypt unregistered [ 6975.661129] Key type ._llcrypt registered [ 6975.662135] Key type .llcrypt registered [ 6975.702711] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 6976.147736] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6976.195958] alg: No test for adler32 (adler32-zlib) [ 6977.063109] Lustre: Lustre: Build Version: 2.17.50_236_gea5aa0c [ 6977.148836] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 6978.743133] Key type lgssc registered [ 6979.062795] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6981.872860] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6981.877456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6982.964209] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6982.975378] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6983.018746] Lustre: lustre-MDT0000: new disk, initializing [ 6983.040353] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6983.046296] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6984.228962] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 6988.132188] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6988.173798] Lustre: 443038:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6988.196167] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6988.197771] Lustre: Skipped 1 previous similar message [ 6988.232532] Lustre: lustre-MDT0001: new disk, initializing [ 6988.262026] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6988.272557] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6988.275978] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6989.415242] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 6992.305891] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6993.362196] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6995.582768] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6995.717777] Lustre: lustre-OST0000: new disk, initializing [ 6995.720084] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6995.754812] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6997.811352] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 6999.366246] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6999.371402] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6999.426400] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7000.846470] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7001.835641] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing lnet_if_list [ 7003.370082] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [ 7003.380726] LNet: Added LNI 192.168.202.131@tcp10 [8/256/0/180] [ 7005.238621] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [ 7005.255086] LNet: Added LNI 192.168.202.131@tcp11 [8/256/0/180] [ 7006.589414] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [ 7006.603529] LNet: Added LNI 192.168.202.131@tcp12 [8/256/0/180] [ 7007.902261] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [ 7007.912028] LNet: Added LNI 192.168.202.131@tcp13 [8/256/0/180] [ 7009.100886] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [ 7009.109386] LNet: Added LNI 192.168.202.131@tcp14 [8/256/0/180] [ 7010.302105] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [ 7010.311448] LNet: Added LNI 192.168.202.131@tcp15 [8/256/0/180] [ 7011.585749] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [ 7011.594627] LNet: Added LNI 192.168.202.131@tcp16 [8/256/0/180] [ 7012.826244] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [ 7012.834882] LNet: Added LNI 192.168.202.131@tcp17 [8/256/0/180] [ 7014.161049] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [ 7014.169596] LNet: Added LNI 192.168.202.131@tcp18 [8/256/0/180] [ 7015.492903] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [ 7015.501730] LNet: Added LNI 192.168.202.131@tcp19 [8/256/0/180] [ 7016.795769] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [ 7016.804476] LNet: Added LNI 192.168.202.131@tcp20 [8/256/0/180] [ 7018.227553] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [ 7018.236441] LNet: Added LNI 192.168.202.131@tcp21 [8/256/0/180] [ 7019.606573] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [ 7019.615235] LNet: Added LNI 192.168.202.131@tcp22 [8/256/0/180] [ 7021.033070] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [ 7021.041995] LNet: Added LNI 192.168.202.131@tcp23 [8/256/0/180] [ 7022.454605] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [ 7022.463661] LNet: Added LNI 192.168.202.131@tcp24 [8/256/0/180] [ 7023.939276] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [ 7023.948354] LNet: Added LNI 192.168.202.131@tcp25 [8/256/0/180] [ 7025.475766] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [ 7025.491678] LNet: Added LNI 192.168.202.131@tcp26 [8/256/0/180] [ 7027.032485] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [ 7027.041445] LNet: Added LNI 192.168.202.131@tcp27 [8/256/0/180] [ 7028.675780] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [ 7028.684514] LNet: Added LNI 192.168.202.131@tcp28 [8/256/0/180] [ 7030.304856] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [ 7030.313437] LNet: Added LNI 192.168.202.131@tcp29 [8/256/0/180] [ 7032.000841] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [ 7032.011469] LNet: Added LNI 192.168.202.131@tcp30 [8/256/0/180] [ 7033.800525] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [ 7033.810342] LNet: Added LNI 192.168.202.131@tcp31 [8/256/0/180] [ 7035.670597] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [ 7035.681453] LNet: Added LNI 192.168.202.131@tcp32 [8/256/0/180] [ 7037.629065] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [ 7037.639486] LNet: Added LNI 192.168.202.131@tcp33 [8/256/0/180] [ 7039.608052] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [ 7039.617349] LNet: Added LNI 192.168.202.131@tcp34 [8/256/0/180] [ 7041.501273] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [ 7041.510502] LNet: Added LNI 192.168.202.131@tcp35 [8/256/0/180] [ 7043.296107] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [ 7043.305447] LNet: Added LNI 192.168.202.131@tcp36 [8/256/0/180] [ 7045.189503] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [ 7045.198146] LNet: Added LNI 192.168.202.131@tcp37 [8/256/0/180] [ 7047.090042] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [ 7047.099329] LNet: Added LNI 192.168.202.131@tcp38 [8/256/0/180] [ 7049.089380] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [ 7049.098610] LNet: Added LNI 192.168.202.131@tcp39 [8/256/0/180] [ 7051.022198] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [ 7051.031220] LNet: Added LNI 192.168.202.131@tcp40 [8/256/0/180] [ 7053.050085] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [ 7053.059241] LNet: Added LNI 192.168.202.131@tcp41 [8/256/0/180] [ 7055.098458] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 7055.109528] LNet: Added LNI 192.168.202.131@tcp42 [8/256/0/180] [ 7057.126211] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [ 7057.135486] LNet: Added LNI 192.168.202.131@tcp43 [8/256/0/180] [ 7059.353367] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [ 7059.363344] LNet: Added LNI 192.168.202.131@tcp44 [8/256/0/180] [ 7061.523209] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [ 7061.534544] LNet: Added LNI 192.168.202.131@tcp45 [8/256/0/180] [ 7063.751995] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [ 7063.761580] LNet: Added LNI 192.168.202.131@tcp46 [8/256/0/180] [ 7066.048773] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [ 7066.059746] LNet: Added LNI 192.168.202.131@tcp47 [8/256/0/180] [ 7068.544790] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [ 7068.554394] LNet: Added LNI 192.168.202.131@tcp48 [8/256/0/180] [ 7071.060153] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [ 7071.071868] LNet: Added LNI 192.168.202.131@tcp49 [8/256/0/180] [ 7073.742135] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [ 7073.752325] LNet: Added LNI 192.168.202.131@tcp50 [8/256/0/180] [ 7081.441494] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7081.442381] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7081.444475] Lustre: Skipped 2 previous similar messages [ 7081.447425] Lustre: Skipped 2 previous similar messages [ 7086.561101] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7086.563949] Lustre: Skipped 2 previous similar messages [ 7086.745315] Lustre: server umount lustre-MDT0000 complete [ 7087.811320] LustreError: 443030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773108918 with bad export cookie 10485797770470594651 [ 7087.814155] LustreError: 443030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7087.814240] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7087.915740] Lustre: server umount lustre-MDT0001 complete [ 7094.437656] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 7096.969586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7097.823858] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x918509841f7ba034 [ 7097.826105] Lustre: MGC192.168.202.131@tcp: Connection restored to 0@lo (at 0@lo) [ 7097.929289] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7099.044191] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 7101.540114] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7101.627895] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7102.771351] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 7104.328531] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7105.382423] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7107.043948] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7108.063153] Lustre: 442312:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773108922/real 1773108922] req@00000000e438d373 x1859239340485632/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1773108938 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7109.087123] Lustre: 442314:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773108923/real 1773108923] req@000000005f08590b x1859239340485760/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1773108939 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7112.161828] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7112.162233] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7112.165687] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7112.167934] Lustre: Skipped 1 previous similar message [ 7117.282559] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7117.284353] Lustre: Skipped 2 previous similar messages [ 7117.968070] Lustre: server umount lustre-OST0000 complete [ 7122.401051] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7122.401465] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7122.404227] Lustre: Skipped 1 previous similar message [ 7122.406949] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7125.273551] Lustre: server umount lustre-MDT0000 complete [ 7126.390994] LustreError: 443030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773108957 with bad export cookie 10485797770470596660 [ 7126.393964] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7126.394313] LustreError: 443030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7126.505329] Lustre: server umount lustre-MDT0001 complete [ 7129.674316] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 7130.621251] Key type lgssc unregistered [ 7130.770763] LNet: 470693:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7130.773360] LNetError: 470693:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7130.775599] LNet: Removed LNI 192.168.202.131@tcp [ 7130.776633] LNet: Removed LNI 192.168.202.131@tcp10 [ 7130.777782] LNet: Removed LNI 192.168.202.131@tcp11 [ 7130.778905] LNet: Removed LNI 192.168.202.131@tcp12 [ 7130.779982] LNet: Removed LNI 192.168.202.131@tcp13 [ 7130.781112] LNet: Removed LNI 192.168.202.131@tcp14 [ 7130.782219] LNet: Removed LNI 192.168.202.131@tcp15 [ 7130.783323] LNet: Removed LNI 192.168.202.131@tcp16 [ 7130.784418] LNet: Removed LNI 192.168.202.131@tcp17 [ 7130.785539] LNet: Removed LNI 192.168.202.131@tcp18 [ 7130.786592] LNet: Removed LNI 192.168.202.131@tcp19 [ 7130.787670] LNet: Removed LNI 192.168.202.131@tcp20 [ 7130.788757] LNet: Removed LNI 192.168.202.131@tcp21 [ 7130.789913] LNet: Removed LNI 192.168.202.131@tcp22 [ 7130.791048] LNet: Removed LNI 192.168.202.131@tcp23 [ 7130.792178] LNet: Removed LNI 192.168.202.131@tcp24 [ 7130.793256] LNet: Removed LNI 192.168.202.131@tcp25 [ 7130.794386] LNet: Removed LNI 192.168.202.131@tcp26 [ 7130.795451] LNet: Removed LNI 192.168.202.131@tcp27 [ 7130.796481] LNet: Removed LNI 192.168.202.131@tcp28 [ 7130.797632] LNet: Removed LNI 192.168.202.131@tcp29 [ 7130.798680] LNet: Removed LNI 192.168.202.131@tcp30 [ 7130.799701] LNet: Removed LNI 192.168.202.131@tcp31 [ 7130.800738] LNet: Removed LNI 192.168.202.131@tcp32 [ 7130.801762] LNet: Removed LNI 192.168.202.131@tcp33 [ 7130.802888] LNet: Removed LNI 192.168.202.131@tcp34 [ 7130.804071] LNet: Removed LNI 192.168.202.131@tcp35 [ 7130.805150] LNet: Removed LNI 192.168.202.131@tcp36 [ 7130.806228] LNet: Removed LNI 192.168.202.131@tcp37 [ 7130.807333] LNet: Removed LNI 192.168.202.131@tcp38 [ 7130.808350] LNet: Removed LNI 192.168.202.131@tcp39 [ 7130.809464] LNet: Removed LNI 192.168.202.131@tcp40 [ 7130.810509] LNet: Removed LNI 192.168.202.131@tcp41 [ 7130.811538] LNet: Removed LNI 192.168.202.131@tcp42 [ 7130.812576] LNet: Removed LNI 192.168.202.131@tcp43 [ 7130.813698] LNet: Removed LNI 192.168.202.131@tcp44 [ 7130.814731] LNet: Removed LNI 192.168.202.131@tcp45 [ 7130.815796] LNet: Removed LNI 192.168.202.131@tcp46 [ 7130.816850] LNet: Removed LNI 192.168.202.131@tcp47 [ 7130.817926] LNet: Removed LNI 192.168.202.131@tcp48 [ 7130.818947] LNet: Removed LNI 192.168.202.131@tcp49 [ 7130.826316] LNet: Removed LNI 192.168.202.131@tcp50 [ 7131.065996] Key type .llcrypt unregistered [ 7131.067163] Key type ._llcrypt unregistered [ 7137.846843] Key type ._llcrypt registered [ 7137.849463] Key type .llcrypt registered [ 7137.882767] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 7138.180477] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7138.214029] alg: No test for adler32 (adler32-zlib) [ 7139.055189] Lustre: Lustre: Build Version: 2.17.50_236_gea5aa0c [ 7139.129044] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 7140.703108] Key type lgssc registered [ 7141.019542] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7143.279170] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7143.284613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7144.440461] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7145.560604] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 7148.057265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7148.169376] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7149.405321] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 7151.053224] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7152.193575] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7154.422173] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7154.537393] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7155.557543] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 7156.244955] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 7157.913866] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7162.785603] loop0: detected capacity change from 0 to 204800000 [ 7162.839235] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7162.842108] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7162.846392] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7164.793463] loop0: detected capacity change from 0 to 204800000 [ 7164.842045] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7164.845255] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7164.849120] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7166.520605] loop0: detected capacity change from 0 to 204800000 [ 7168.223910] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7168.278105] Lustre: fs15246-OST0000: new disk, initializing [ 7168.279959] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 7168.299170] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7170.024829] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 7173.585732] loop1: detected capacity change from 0 to 204800000 [ 7175.295470] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7175.335417] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 7175.354147] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 7175.388312] Lustre: fs15246-MDT0000: new disk, initializing [ 7175.432838] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7175.442853] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7175.478345] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 7176.543772] Lustre: 476274:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000f8e149e7 x1859239510307328/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1773109023 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 7176.575261] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 7185.631200] LustreError: 476274:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 7185.634113] LustreError: 476274:0:(fid_request.c:255:seq_client_get_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Can't allocate new sequence: rc = -5 [ 7185.636726] LustreError: 476274:0:(osp_precreate.c:459:osp_precreate_rollover_new_seq()) fs15246-OST0000-osc-MDT0000: alloc fid error: rc = -5 [ 7187.939830] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7187.943168] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 7189.262835] Lustre: server umount fs15246-MDT0000 complete [ 7192.017976] Lustre: server umount fs15246-OST0000 complete [ 7196.639726] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7196.641826] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7196.645438] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7198.176529] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7198.178302] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7201.116420] Lustre: server umount lustre-OST0000 complete [ 7203.296521] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7203.301826] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7203.303721] Lustre: Skipped 2 previous similar messages [ 7208.416372] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7208.418128] Lustre: Skipped 1 previous similar message [ 7208.796664] Lustre: server umount lustre-MDT0000 complete [ 7210.561898] LustreError: 471896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773109041 with bad export cookie 27665695848022881 [ 7210.568645] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7210.569159] LustreError: 471896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7210.721221] Lustre: server umount lustre-MDT0001 complete [ 7214.832110] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 7216.187544] Key type lgssc unregistered [ 7216.345689] LNet: 479334:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7216.350923] LNetError: 479334:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7216.361587] LNet: Removed LNI 192.168.202.131@tcp [ 7216.817727] Key type .llcrypt unregistered [ 7216.819409] Key type ._llcrypt unregistered [ 7225.521006] Key type ._llcrypt registered [ 7225.522828] Key type .llcrypt registered [ 7225.563770] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 7225.915671] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7225.944413] alg: No test for adler32 (adler32-zlib) [ 7226.805765] Lustre: Lustre: Build Version: 2.17.50_236_gea5aa0c [ 7226.896130] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 7228.479172] Key type lgssc registered [ 7229.042248] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7246.770553] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_hostid [ 7250.523605] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 7266.441294] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7266.447585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7267.592160] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7267.609772] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7267.660349] Lustre: lustre-MDT0000: new disk, initializing [ 7267.697059] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7267.709325] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7269.708454] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 7275.105363] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7275.185564] Lustre: 483722:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7275.233852] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7275.237458] Lustre: Skipped 1 previous similar message [ 7275.291372] Lustre: lustre-MDT0001: new disk, initializing [ 7275.355290] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7275.378506] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7275.386476] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7277.220521] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 7281.236458] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7282.827685] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7286.136666] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7286.319157] Lustre: lustre-OST0000: new disk, initializing [ 7286.324266] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7286.374253] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7288.660499] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 7292.323522] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7294.173676] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7296.025064] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7296.031367] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7296.095796] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7296.370930] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 7298.380761] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7298.518175] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7301.092041] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7301.094045] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7301.098182] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7301.102846] Lustre: Skipped 1 previous similar message [ 7305.571715] Lustre: server umount lustre-OST0000 complete [ 7311.329086] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7311.329103] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7311.329930] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7311.332948] Lustre: Skipped 2 previous similar messages [ 7311.345842] LustreError: Skipped 1 previous similar message [ 7313.329909] Lustre: server umount lustre-MDT0000 complete [ 7315.125070] LustreError: 484501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773109145 with bad export cookie 2836522646388255450 [ 7315.132300] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7315.136151] LustreError: 484501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7315.293826] Lustre: server umount lustre-MDT0001 complete [ 7326.356019] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_hostid [ 7330.695062] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 7345.622434] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7178 sec ========== 22:19:36 (1773109176) [ 7346.379491] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 22:19:37 (1773109177) === [ 7347.868881] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 22:19:38 (1773109178) === [ 7365.501023] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 7366.986810] Key type lgssc unregistered [ 7367.156836] LNet: 491927:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7367.161497] LNetError: 491927:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7367.172680] LNet: Removed LNI 192.168.202.131@tcp [ 7367.645919] Key type .llcrypt unregistered [ 7367.648286] Key type ._llcrypt unregistered