[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 507774967 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2640MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002321] x2apic enabled [ 0.003010] Switched APIC routing to physical x2apic. [ 0.004017] kvm-guest: setup PV IPIs [ 0.007347] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008018] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009013] pid_max: default: 32768 minimum: 301 [ 0.011141] LSM: Security Framework initializing [ 0.012058] Yama: becoming mindful. [ 0.013045] SELinux: Initializing. [ 0.014086] *** VALIDATE selinux *** [ 0.022561] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026859] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027176] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028120] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029125] *** VALIDATE tmpfs *** [ 0.031352] *** VALIDATE proc *** [ 0.032276] *** VALIDATE cgroup *** [ 0.033011] *** VALIDATE cgroup2 *** [ 0.034298] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035175] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037031] Spectre V2 : User space: Vulnerable [ 0.039006] Speculative Store Bypass: Vulnerable [ 0.042215] debug: unmapping init [mem 0xffffffffad259000-0xffffffffad260fff] [ 0.044175] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045668] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046023] ... version: 2 [ 0.047014] ... bit width: 48 [ 0.048010] ... generic registers: 4 [ 0.049011] ... value mask: 0000ffffffffffff [ 0.050017] ... max period: 00007fffffffffff [ 0.051016] ... fixed-purpose events: 3 [ 0.052017] ... event mask: 000000070000000f [ 0.053330] rcu: Hierarchical SRCU implementation. [ 0.055505] smp: Bringing up secondary CPUs ... [ 0.056597] x86: Booting SMP configuration: [ 0.057035] .... node #0, CPUs: #1 #2 #3 [ 0.060503] smp: Brought up 1 node, 4 CPUs [ 0.062014] smpboot: Max logical packages: 1 [ 0.063021] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.135332] node 0 deferred pages initialised in 70ms [ 0.138137] devtmpfs: initialized [ 0.139279] x86/mm: Memory block size: 128MB [ 0.143007] gcov: version magic: 0x41383552 [ 0.146224] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.147084] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.148343] pinctrl core: initialized pinctrl subsystem [ 0.149210] [ 0.149829] ************************************************************* [ 0.150018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.151011] ** ** [ 0.152018] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.153020] ** ** [ 0.154018] ** This means that this kernel is built to expose internal ** [ 0.155014] ** IOMMU data structures, which may compromise security on ** [ 0.156014] ** your system. ** [ 0.157015] ** ** [ 0.158016] ** If you see this message and you are not debugging the ** [ 0.159015] ** kernel, report this immediately to your vendor! ** [ 0.160015] ** ** [ 0.161014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.162014] ************************************************************* [ 0.163529] NET: Registered protocol family 16 [ 0.165514] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.168072] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.171072] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.175543] cpuidle: using governor menu [ 0.177752] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.179477] PCI: Using configuration type 1 for base access [ 0.181193] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.190067] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.192030] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.195034] cryptd: max_cpu_qlen set to 1000 [ 0.197183] ACPI: Added _OSI(Module Device) [ 0.198013] ACPI: Added _OSI(Processor Device) [ 0.198943] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.200011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.203417] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.209273] ACPI: Interpreter enabled [ 0.210041] ACPI: PM: (supports S0 S3 S4 S5) [ 0.211012] ACPI: Using IOAPIC for interrupt routing [ 0.212096] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.214354] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.221653] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.223035] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.225015] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.227092] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.230056] acpiphp: Slot [2] registered [ 0.231078] acpiphp: Slot [5] registered [ 0.232105] acpiphp: Slot [6] registered [ 0.233080] acpiphp: Slot [7] registered [ 0.233964] acpiphp: Slot [8] registered [ 0.235062] acpiphp: Slot [9] registered [ 0.235995] acpiphp: Slot [10] registered [ 0.237068] acpiphp: Slot [3] registered [ 0.237954] acpiphp: Slot [4] registered [ 0.239102] acpiphp: Slot [11] registered [ 0.240104] acpiphp: Slot [12] registered [ 0.241094] acpiphp: Slot [13] registered [ 0.241980] acpiphp: Slot [14] registered [ 0.243076] acpiphp: Slot [15] registered [ 0.243950] acpiphp: Slot [16] registered [ 0.245092] acpiphp: Slot [17] registered [ 0.245983] acpiphp: Slot [18] registered [ 0.247056] acpiphp: Slot [19] registered [ 0.247992] acpiphp: Slot [20] registered [ 0.249086] acpiphp: Slot [21] registered [ 0.250122] acpiphp: Slot [22] registered [ 0.252138] acpiphp: Slot [23] registered [ 0.253131] acpiphp: Slot [24] registered [ 0.255135] acpiphp: Slot [25] registered [ 0.256099] acpiphp: Slot [26] registered [ 0.258120] acpiphp: Slot [27] registered [ 0.259136] acpiphp: Slot [28] registered [ 0.261101] acpiphp: Slot [29] registered [ 0.262129] acpiphp: Slot [30] registered [ 0.264101] acpiphp: Slot [31] registered [ 0.265048] PCI host bridge to bus 0000:00 [ 0.266016] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.268023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.270019] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.272018] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.273018] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.275025] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.277198] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.280012] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.282966] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.291017] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.295056] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.297014] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.298014] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.300013] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.302576] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.304773] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.308047] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.310909] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.315018] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.328016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.332021] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.337073] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.390015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.398017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.415019] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.423312] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.429025] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.437017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.460020] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.468950] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.492020] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.511014] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.534014] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.541140] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.574022] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.583070] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.595027] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.603992] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.610018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.615020] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.628015] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.639576] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.643025] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.649017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.659021] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.669108] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.670310] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.673357] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.675302] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.677272] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.681090] iommu: Default domain type: Passthrough [ 0.683422] SCSI subsystem initialized [ 0.684079] ACPI: bus type USB registered [ 0.685087] usbcore: registered new interface driver usbfs [ 0.687088] usbcore: registered new interface driver hub [ 0.688079] usbcore: registered new device driver usb [ 0.689173] pps_core: LinuxPPS API ver. 1 registered [ 0.691010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.693054] PTP clock support registered [ 0.694144] EDAC MC: Ver: 3.0.0 [ 0.696746] PCI: Using ACPI for IRQ routing [ 0.698966] NetLabel: Initializing [ 0.700014] NetLabel: domain hash size = 128 [ 0.701076] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.704099] NetLabel: unlabeled traffic allowed by default [ 0.706353] vgaarb: loaded [ 0.708273] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.710016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.718131] clocksource: Switched to clocksource kvm-clock [ 0.819700] VFS: Disk quotas dquot_6.6.0 [ 0.821325] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.824179] *** VALIDATE ramfs *** [ 0.825473] *** VALIDATE hugetlbfs *** [ 0.827181] pnp: PnP ACPI init [ 0.829605] pnp: PnP ACPI: found 6 devices [ 0.845567] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.848859] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.851245] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.853523] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.856193] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.858886] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.862046] NET: Registered protocol family 2 [ 0.864167] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.869077] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.873038] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.878632] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.882498] TCP: Hash tables configured (established 65536 bind 65536) [ 0.885806] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.888898] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.891801] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.894560] NET: Registered protocol family 1 [ 0.896991] RPC: Registered named UNIX socket transport module. [ 0.899459] RPC: Registered udp transport module. [ 0.901459] RPC: Registered tcp transport module. [ 0.903457] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.906197] NET: Registered protocol family 44 [ 0.907709] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.909958] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.912039] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.914106] PCI: CLS 0 bytes, default 64 [ 0.915733] Unpacking initramfs... [ 2.320306] debug: unmapping init [mem 0xffff96317cc54000-0xffff96317ffbffff] [ 2.324361] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.326423] software IO TLB: mapped [mem 0x00000000a1000000-0x00000000a5000000] (64MB) [ 2.329081] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.840189] Initialise system trusted keyrings [ 2.841783] Key type blacklist registered [ 2.843643] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.852620] zbud: loaded [ 2.855899] *** VALIDATE nfs *** [ 2.857303] *** VALIDATE nfs4 *** [ 2.859127] pstore: using deflate compression [ 2.863062] Platform Keyring initialized [ 2.973844] NET: Registered protocol family 38 [ 2.975486] Key type asymmetric registered [ 2.976946] Asymmetric key parser 'x509' registered [ 2.979016] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.982125] io scheduler mq-deadline registered [ 2.983513] io scheduler kyber registered [ 2.985127] io scheduler bfq registered [ 2.986864] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.990076] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.993097] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.996134] ACPI: Power Button [PWRF] [ 3.001145] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.008058] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.020514] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.038387] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.054117] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.084058] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.114858] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.119171] Non-volatile memory driver v1.3 [ 3.120722] Linux agpgart interface v0.103 [ 3.155511] virtio_blk virtio1: [vda] 149976 512-byte logical blocks (76.8 MB/73.2 MiB) [ 3.158588] vda: detected capacity change from 0 to 76787712 [ 3.179547] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.182655] vdb: detected capacity change from 0 to 1073741824 [ 3.204667] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.207557] vdc: detected capacity change from 0 to 2621440000 [ 3.224582] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.228089] vdd: detected capacity change from 0 to 2621440000 [ 3.246622] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.249895] vde: detected capacity change from 0 to 4294967296 [ 3.265954] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.269071] vdf: detected capacity change from 0 to 4294967296 [ 3.280414] libphy: Fixed MDIO Bus: probed [ 3.286727] usbcore: registered new interface driver usbserial_generic [ 3.289531] usbserial: USB Serial support registered for generic [ 3.292177] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.296858] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.298893] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.301348] mousedev: PS/2 mouse device common for all mice [ 3.304407] rtc_cmos 00:05: RTC can wake from S4 [ 3.310170] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.311126] rtc_cmos 00:05: registered as rtc0 [ 3.318769] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.323439] intel_pstate: CPU model not supported [ 3.327544] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.330294] hid: raw HID events driver (C) Jiri Kosina [ 3.336850] usbcore: registered new interface driver usbhid [ 3.338249] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.339468] usbhid: USB HID core driver [ 3.344533] drop_monitor: Initializing network drop monitor service [ 3.347071] Initializing XFRM netlink socket [ 3.349369] NET: Registered protocol family 10 [ 3.352512] Segment Routing with IPv6 [ 3.353969] NET: Registered protocol family 17 [ 3.355960] mpls_gso: MPLS GSO support [ 3.361416] RAS: Correctable Errors collector initialized. [ 3.364037] AVX version of gcm_enc/dec engaged. [ 3.365802] AES CTR mode by8 optimization enabled [ 3.445515] sched_clock: Marking stable (3445490608, 0)->(4368778795, -923288187) [ 3.448929] registered taskstats version 1 [ 3.450530] Loading compiled-in X.509 certificates [ 3.452806] zswap: loaded using pool lzo/zbud [ 3.476048] Key type big_key registered [ 3.490008] Key type encrypted registered [ 3.491774] ima: No TPM chip found, activating TPM-bypass! [ 3.493960] ima: Allocated hash algorithm: sha1 [ 3.495844] ima: No architecture policies found [ 3.497790] evm: Initialising EVM extended attributes: [ 3.499760] evm: security.selinux [ 3.501103] evm: security.ima [ 3.502271] evm: security.capability [ 3.503670] evm: HMAC attrs: 0x1 [ 3.506356] rtc_cmos 00:05: setting system clock to 2026-09-07 20:42:05 UTC (1788813725) [ 3.513179] debug: unmapping init [mem 0xffffffffae203000-0xffffffffae3fffff] [ 3.516656] debug: unmapping init [mem 0xffffffffacf82000-0xffffffffad258fff] [ 3.525162] Write protecting the kernel read-only data: 28672k [ 3.528952] debug: unmapping init [mem 0xffffffffab603000-0xffffffffab7fffff] [ 3.531971] debug: unmapping init [mem 0xffffffffabf14000-0xffffffffabffffff] [ 3.567080] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.576135] systemd[1]: Detected virtualization kvm. [ 3.578205] systemd[1]: Detected architecture x86-64. [ 3.580197] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.604064] systemd[1]: No hostname configured. [ 3.605955] systemd[1]: Set hostname to . [ 3.608285] random: systemd: uninitialized urandom read (16 bytes read) [ 3.610752] systemd[1]: Initializing machine ID from random generator. [ 3.663546] random: ln: uninitialized urandom read (6 bytes read) [ 3.753270] random: systemd: uninitialized urandom read (16 bytes read) [ 3.756042] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.761030] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.765794] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Swap. [ OK ] Reached target Local File Systems. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... [ OK ] Reached target Timers. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.301619] device-mapper: uevent: version 1.0.3 [ 4.303068] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.909580] virtio_net virtio0 ens2: renamed from eth0 [ 4.927129] random: fast init done [ 4.984181] scsi host0: ata_piix [ 5.003691] scsi host1: ata_piix [ 5.006176] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.009831] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.180852] dracut-initqueue[591]: RTNETLINK answers: File exists [ 10.008986] random: crng init done [ 10.009828] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.213953] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.292821] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.553227] SELinux: Disabled at runtime. [ 11.618723] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.627627] systemd[1]: Detected virtualization kvm. [ 11.629597] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.107954] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.110979] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.114597] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.117466] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.122501] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.130334] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.134995] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target rpc_pipefs.target. Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Activating swap /dev/disk/by-label/SWAP... Starting Remount Root and Kernel File Systems... [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Huge Pages File System... Starting Create list of required st…ce nodes for the current kernel... [ 12.202118] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Kernel Debug File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Mounting POSIX Message Queue File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Journal Service. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.522895] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.837607] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.915790] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.973213] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.000943] EDAC sbridge: Ver: 1.1.2 [ 14.401963] Key type dns_resolver registered [ 14.685982] NFS: Registering the id_resolver key type [ 14.688045] Key type id_resolver registered [ 14.689549] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Authorization Manager. [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg239-server login: [ 37.325005] hrtimer: interrupt took 2442471 ns [ 59.405455] libcfs: loading out-of-tree module taints kernel. [ 59.472429] Key type ._llcrypt registered [ 59.474827] Key type .llcrypt registered [ 59.607186] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_hostid [ 81.931614] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 83.610886] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 83.625456] alg: No test for adler32 (adler32-zlib) [ 84.986237] Lustre: Lustre: Build Version: 2.17.58_40_gea5fd3f [ 85.720670] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [ 87.560308] Key type lgssc registered [ 89.480178] Lustre: Echo OBD driver; http://www.lustre.org/ [ 109.088720] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 152.069337] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 165.426374] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 165.463738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 166.907812] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 166.940731] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 167.094288] Lustre: lustre-MDT0000: new disk, initializing [ 167.173725] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 167.188221] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 172.223519] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 187.134590] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 187.294149] Lustre: 6523:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 187.373795] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 187.379990] Lustre: Skipped 1 previous similar message [ 187.495541] Lustre: lustre-MDT0001: new disk, initializing [ 187.572427] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 187.641893] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 187.670720] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 192.327513] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 197.625345] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 207.197468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 207.560184] Lustre: lustre-OST0000: new disk, initializing [ 207.564043] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 207.568831] Lustre: 8461:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 207.625551] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 214.047393] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 214.057038] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 214.107898] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 214.395616] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 228.065213] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 228.186460] Lustre: lustre-OST0001: new disk, initializing [ 228.189530] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 228.194599] Lustre: 9534:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 228.237098] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 234.558805] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 234.569312] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 234.570719] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 234.615814] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 246.001900] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 251.423641] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 257.641416] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing check_logdir /tmp/testlogs/ [ 262.590738] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing yml_node [ 266.346326] Lustre: DEBUG MARKER: Client: 2.17.58.40 [ 268.903308] Lustre: DEBUG MARKER: MDS: 2.17.58.40 [ 271.087951] Lustre: DEBUG MARKER: OSS: 2.17.58.40 [ 272.734511] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Sep 7 16:46:33 EDT 2026 [ 289.133357] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 290.093875] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 305.634068] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 305.642908] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 305.653543] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 306.147319] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 306.176790] Lustre: Skipped 1 previous similar message [ 306.184160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 306.186590] Lustre: Skipped 2 previous similar messages [ 310.165907] Lustre: server umount lustre-MDT0000 complete [ 313.228406] LustreError: 6515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788814035 with bad export cookie 5856200018049223742 [ 313.228750] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 313.240063] LustreError: 6515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 316.394793] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 316.399170] LustreError: 6528:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 316.416334] Lustre: Skipped 1 previous similar message [ 316.418075] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 316.447161] LustreError: 6528:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 318.432848] LustreError: 8086:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 318.432866] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 318.446188] LustreError: 8086:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 318.449064] Lustre: Skipped 2 previous similar messages [ 319.735259] Lustre: server umount lustre-MDT0001 complete [ 324.000844] Lustre: server umount lustre-OST0000 complete [ 327.866854] Lustre: server umount lustre-OST0001 complete [ 334.396656] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_hostid [ 343.301511] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 380.246052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 380.458076] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 380.485495] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 380.579424] Lustre: lustre-MDT0000: new disk, initializing [ 380.646507] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 380.662519] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 385.622902] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 396.142408] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 396.310374] Lustre: 15083:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 396.369709] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 396.374769] Lustre: Skipped 1 previous similar message [ 396.487653] Lustre: lustre-MDT0001: new disk, initializing [ 396.611903] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 396.656746] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 396.675402] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 401.907696] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 411.947646] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 418.344733] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 424.996571] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 425.253815] Lustre: lustre-OST0000: new disk, initializing [ 425.257815] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 425.263887] Lustre: 17054:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 425.373421] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 426.891676] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 426.918243] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 427.082652] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 431.469741] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 440.248139] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 444.508788] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 444.746687] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 449.181540] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 449.385197] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 452.577809] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 452.583626] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 452.589655] LustreError: Skipped 1 previous similar message [ 452.594664] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 452.605299] Lustre: Skipped 2 previous similar messages [ 452.623610] Lustre: Skipped 1 previous similar message [ 462.819537] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 462.829934] Lustre: Skipped 3 previous similar messages [ 465.376183] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 465.492821] Lustre: server umount lustre-OST0000 complete [ 473.059380] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 473.076345] Lustre: Skipped 1 previous similar message [ 483.296133] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 483.302206] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 483.313451] Lustre: Skipped 5 previous similar messages [ 483.584986] Lustre: server umount lustre-MDT0000 complete [ 487.615257] LustreError: 17053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788814209 with bad export cookie 5856200018049225275 [ 487.621363] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 487.627285] LustreError: 17053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 488.194825] Lustre: server umount lustre-MDT0001 complete [ 495.139919] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 16:50:15 (1788814215) [ 514.774805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 515.374682] LustreError: 19931:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 515.467393] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 520.200368] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 520.679268] LustreError: 19932:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 525.804622] LustreError: 19931:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 529.198070] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 534.422370] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 542.056975] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 548.484554] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 550.584431] Lustre: Failing over lustre-MDT0000 [ 550.886406] Lustre: server umount lustre-MDT0000 complete [ 554.724167] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 554.742845] LustreError: 20659:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 554.747840] Lustre: Skipped 1 previous similar message [ 554.770297] LustreError: 20659:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 570.336180] Lustre: 3655:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788814276/real 1788814276] req@ffff9630c400f800 x1875707226644480/t0(0) o400->MGC192.168.202.139@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788814292 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 570.379607] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 570.653250] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 574.497782] LustreError: 19931:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 574.505498] LustreError: 19931:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 581.601573] LustreError: 3652:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9630c3ae8380 x1875707226647424/t0(0) o250->MGC192.168.202.139@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 581.918205] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 581.928927] Lustre: Skipped 1 previous similar message [ 586.693339] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 587.236537] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 587.242601] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 587.261844] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 596.226236] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 602.596190] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 602.604916] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 602.623479] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 615.392634] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 615.716787] Lustre: server umount lustre-MDT0000 complete [ 619.575940] LustreError: 19912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788814341 with bad export cookie 5856200018049226689 [ 619.596929] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 619.602737] LustreError: 19912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 620.000463] Lustre: server umount lustre-MDT0001 complete [ 628.997810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 629.486657] LustreError: 23268:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 629.503879] LustreError: 23268:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 634.284934] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 643.500169] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 648.044179] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 655.578671] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 660.347818] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 662.459593] Lustre: Failing over lustre-MDT0000 [ 662.728862] Lustre: server umount lustre-MDT0000 complete [ 664.033781] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 664.055109] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 664.064971] Lustre: Skipped 1 previous similar message [ 680.798865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 680.917737] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 680.925817] Lustre: 3653:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788814386/real 1788814386] req@ffff9631fbf0bb80 x1875707226674944/t0(0) o400->MGC192.168.202.139@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788814402 ref 1 fl Rpc:EXNQr/200/ffffffff rc -5/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 681.191438] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 685.279904] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 686.568114] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 686.575471] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 686.586296] Lustre: Skipped 1 previous similar message [ 686.604300] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 694.312922] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 701.927798] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 701.937697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 701.946380] Lustre: Skipped 6 previous similar messages [ 704.656843] Lustre: server umount lustre-MDT0000 complete [ 707.556081] LustreError: 25757:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 707.572132] LustreError: 25757:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [ 708.753541] LustreError: 23251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788814430 with bad export cookie 5856200018049227627 [ 708.758677] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 708.768970] LustreError: 23251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 709.142754] Lustre: server umount lustre-MDT0001 complete [ 717.115558] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 16:53:57 (1788814437) [ 733.071610] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_hostid [ 742.588603] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 780.624349] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 791.657573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 791.975696] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 792.030542] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 792.176971] Lustre: lustre-MDT0000: new disk, initializing [ 792.274309] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 792.280788] Lustre: Skipped 3 previous similar messages [ 792.305134] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 797.005874] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 806.405717] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 806.528792] Lustre: 29642:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 806.829782] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 811.425244] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 820.453818] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 826.270234] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 833.906251] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 834.401039] Lustre: 31608:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 835.774681] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 835.840293] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 835.951762] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 835.972255] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 842.816478] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 853.927225] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 860.262663] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 882.150137] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 882.166371] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 882.167637] LustreError: Skipped 1 previous similar message [ 882.183276] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 882.189367] Lustre: Skipped 4 previous similar messages [ 882.209417] Lustre: Skipped 3 previous similar messages [ 887.064397] Lustre: server umount lustre-OST0000 complete [ 897.508733] LustreError: 29652:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 897.601545] Lustre: server umount lustre-MDT0000 complete [ 901.409441] LustreError: 29633:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788814623 with bad export cookie 5856200018049228033 [ 901.432556] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 901.436884] LustreError: 29633:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 901.968580] Lustre: server umount lustre-MDT0001 complete [ 913.277531] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [ 915.762640] Key type lgssc unregistered [ 916.064176] LNet: 33591:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 916.074296] LNetError: 33591:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 916.085643] LNet: Removed LNI 192.168.202.139@tcp [ 916.940259] Key type .llcrypt unregistered [ 916.948403] Key type ._llcrypt unregistered [ 939.686162] Key type ._llcrypt registered [ 939.691957] Key type .llcrypt registered [ 939.823365] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 940.844276] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 940.947137] alg: No test for adler32 (adler32-zlib) [ 942.111747] Lustre: Lustre: Build Version: 2.17.58_40_gea5fd3f [ 942.495438] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [ 944.160271] Key type lgssc registered [ 945.241331] Lustre: Echo OBD driver; http://www.lustre.org/ [ 952.627243] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 952.662253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 954.159875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 959.041445] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 964.583298] LustreError: 35016:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 964.604304] LustreError: 35016:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 967.795276] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 968.400367] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 972.538283] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 980.208502] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 985.964798] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 993.020309] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 993.191717] LustreError: 36838:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 993.316490] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 998.396072] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 999.450110] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 999.818957] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 1006.634191] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1020.717916] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 16:59:01 (1788814741) [ 1022.540986] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1024.306697] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1030.333640] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 16:59:11 (1788814751) [ 1039.147877] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 16:59:19 (1788814759) [ 1040.118171] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1048.848302] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1048.851709] Lustre: Skipped 2 previous similar messages [ 1050.355186] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 16:59:31 (1788814771) [ 1053.762930] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1053.765987] Lustre: Skipped 3 previous similar messages [ 1071.903611] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1071.907640] Lustre: Skipped 6 previous similar messages [ 1074.322647] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 16:59:55 (1788814795) [ 1077.302802] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1081.049595] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1091.261890] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 17:00:11 (1788814811) [ 1092.962708] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1092.967536] Lustre: Skipped 3 previous similar messages [ 1108.600751] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 17:00:28 (1788814828) [ 1126.371677] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1126.385185] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1126.401464] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1127.395983] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1127.399919] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1127.411802] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1132.521779] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1132.530050] Lustre: Skipped 2 previous similar messages [ 1137.641353] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1137.645374] Lustre: Skipped 1 previous similar message [ 1138.144324] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1138.329906] Lustre: server umount lustre-OST0000 complete [ 1147.361477] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1147.376217] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1147.394242] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1149.582130] Lustre: server umount lustre-MDT0000 complete [ 1152.998252] LustreError: 35013:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1153.029927] LustreError: 35013:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1153.864705] LustreError: 35751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788814875 with bad export cookie 248369888441139899 [ 1153.871700] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1153.875406] LustreError: 35751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1154.175604] Lustre: server umount lustre-MDT0001 complete [ 1166.137437] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [ 1169.095563] Key type lgssc unregistered [ 1169.477827] LNet: 42883:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1169.485801] LNetError: 42883:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1169.517532] LNet: Removed LNI 192.168.202.139@tcp [ 1170.479688] Key type .llcrypt unregistered [ 1170.485629] Key type ._llcrypt unregistered [ 1190.481484] Key type ._llcrypt registered [ 1190.483608] Key type .llcrypt registered [ 1190.574385] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_hostid [ 1209.042656] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 1209.950331] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1210.034782] alg: No test for adler32 (adler32-zlib) [ 1211.041316] Lustre: Lustre: Build Version: 2.17.58_40_gea5fd3f [ 1211.277624] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [ 1212.960535] Key type lgssc registered [ 1214.264066] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1250.552602] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1250.595149] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1251.845858] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1251.880672] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1251.980683] Lustre: lustre-MDT0000: new disk, initializing [ 1252.072810] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1252.105844] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1257.082835] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 1267.932039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1267.986701] Lustre: 46033:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1268.022624] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1268.027460] Lustre: Skipped 1 previous similar message [ 1268.118191] Lustre: lustre-MDT0001: new disk, initializing [ 1268.183606] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1268.205788] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1268.214829] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1272.568448] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 1281.973902] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1286.890736] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1292.334112] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1292.554799] Lustre: lustre-OST0000: new disk, initializing [ 1292.558448] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1292.562860] Lustre: 48003:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1292.637230] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1297.739845] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 1297.976144] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1297.987372] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1298.071969] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1305.820366] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1309.795760] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1310.010982] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1314.369735] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1314.604512] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1318.373102] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1318.382696] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1318.397044] Lustre: Skipped 1 previous similar message [ 1318.408373] Lustre: Skipped 1 previous similar message [ 1322.757789] Lustre: server umount lustre-OST0000 complete [ 1328.611525] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1328.625367] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1328.630270] Lustre: Skipped 1 previous similar message [ 1332.873129] Lustre: server umount lustre-MDT0000 complete [ 1336.721795] LustreError: 46026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788815058 with bad export cookie 11553278004630935399 [ 1336.726786] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1336.731911] LustreError: 46026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1337.109205] Lustre: server umount lustre-MDT0001 complete [ 1346.807859] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 17:04:27 (1788815067) [ 1353.309354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1353.793413] LustreError: 50463:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1353.823281] LustreError: 50463:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 1353.900808] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1358.726538] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 1359.333052] LustreError: 50464:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1363.426977] LustreError: 50463:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1367.492387] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1367.890724] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1372.090438] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 1378.618147] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1384.237960] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1390.247960] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1390.547235] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1396.095335] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 1402.220917] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1406.675483] LustreError: 52990:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1407.730981] Lustre: Modifying parameter general.timeout=1 in log params [ 1408.770638] Lustre: Modifying parameter general.timeout=2 in log params [ 1411.292297] Lustre: Modifying parameter general.timeout=5 in log params [ 1411.295031] Lustre: Skipped 2 previous similar messages [ 1415.985864] Lustre: Modifying parameter general.timeout=11 in log params [ 1415.990251] Lustre: Skipped 5 previous similar messages [ 1424.442911] Lustre: Modifying parameter general.timeout=22 in log params [ 1424.446743] Lustre: Skipped 10 previous similar messages [ 1440.822188] Lustre: Modifying parameter general.timeout=46 in log params [ 1440.824712] Lustre: Skipped 23 previous similar messages [ 1473.187151] Lustre: Modifying parameter general.timeout=88 in log params [ 1473.190281] Lustre: Skipped 41 previous similar messages [ 1518.359686] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 17:07:19 (1788815239) [ 1520.511480] Lustre: 59917:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1522.327634] Lustre: 60025:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1525.494445] Lustre: 60226:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1525.502384] Lustre: 60226:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1538.969892] Lustre: Disabling parameter general.ost.OSS.ost_io.nrs_tbf_rule=start rule1 uid={500 502 } rate=1000 in log params [ 1538.976858] Lustre: Skipped 57 previous similar messages [ 1551.042194] Lustre: 61798:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1551.048980] Lustre: 61798:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1551.066855] LustreError: 61798:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1554.615485] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 17:07:55 (1788815275) [ 1567.713477] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1567.730501] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1567.749735] Lustre: Skipped 1 previous similar message [ 1567.772444] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1571.324708] Lustre: server umount lustre-MDT0000 complete [ 1573.348341] LustreError: 50459:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1573.367081] LustreError: 50459:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1574.243609] LustreError: 53041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788815296 with bad export cookie 11553278004630936281 [ 1574.250957] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1574.456287] Lustre: server umount lustre-MDT0001 complete [ 1588.153759] Lustre: server umount lustre-OST0000 complete [ 1611.329053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1611.556677] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1611.723990] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1615.224467] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 1622.929470] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1623.009140] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1623.043652] Lustre: 64330:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1627.359473] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 1633.785443] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1638.848906] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1644.758758] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1644.972654] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1645.186300] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1645.192361] Lustre: Skipped 1 previous similar message [ 1650.039136] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 1655.785673] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1657.069129] Lustre: 66895:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1662.947598] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1662.949340] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1662.956877] LustreError: Skipped 1 previous similar message [ 1662.984731] Lustre: Skipped 2 previous similar messages [ 1662.987030] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1663.006733] Lustre: Skipped 4 previous similar messages [ 1668.073779] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1668.084860] Lustre: Skipped 1 previous similar message [ 1668.926903] Lustre: server umount lustre-OST0000 complete [ 1673.188788] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1673.203042] Lustre: Skipped 1 previous similar message [ 1678.307907] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1678.316176] Lustre: Skipped 2 previous similar messages [ 1678.395723] Lustre: server umount lustre-MDT0000 complete [ 1680.698714] LustreError: 64322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788815402 with bad export cookie 11553278004630939158 [ 1680.703993] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1680.707529] LustreError: 64322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 1680.902343] Lustre: server umount lustre-MDT0001 complete [ 1689.930762] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [ 1692.261613] Key type lgssc unregistered [ 1692.500164] LNet: 68101:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1692.505212] LNetError: 68101:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1693.543673] LNet: Removed LNI 192.168.202.139@tcp [ 1694.287743] Key type .llcrypt unregistered [ 1694.290219] Key type ._llcrypt unregistered [ 1716.117915] Key type ._llcrypt registered [ 1716.120376] Key type .llcrypt registered [ 1716.218611] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 1716.974918] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1717.129328] alg: No test for adler32 (adler32-zlib) [ 1718.199070] Lustre: Lustre: Build Version: 2.17.58_40_gea5fd3f [ 1718.443694] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [ 1720.080227] Key type lgssc registered [ 1721.018878] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1727.500794] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1727.534892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1729.057774] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1733.034671] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 1738.212802] LustreError: 69490:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1738.227112] LustreError: 69490:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1739.928205] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1740.408987] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1743.982749] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 1749.751101] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1753.686163] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1758.931471] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1759.363933] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1764.396768] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 1770.432092] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1781.991975] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1789.411036] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1789.413968] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1789.417912] Lustre: Skipped 2 previous similar messages [ 1789.425708] Lustre: Skipped 2 previous similar messages [ 1793.963493] Lustre: server umount lustre-MDT0000 complete [ 1794.529713] LustreError: 70221:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1794.544725] LustreError: 70221:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1796.633614] LustreError: 70232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788815518 with bad export cookie 4618703408085795492 [ 1796.637746] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1796.638908] LustreError: 70232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1796.900902] Lustre: server umount lustre-MDT0001 complete [ 1810.035976] Lustre: server umount lustre-OST0000 complete [ 1830.939822] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1831.151117] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1831.171733] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1831.381597] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1834.734783] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 1841.567238] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1841.626689] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1841.640253] Lustre: 74354:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1841.983807] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1845.195393] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 1850.458303] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1854.709347] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1859.663878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1859.791800] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1860.023682] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1864.404719] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 1869.155793] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1870.269401] Lustre: 76930:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1871.710292] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1876.452113] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1876.458071] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1876.463133] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1876.472980] Lustre: Skipped 1 previous similar message [ 1881.452699] Lustre: server umount lustre-OST0000 complete [ 1886.690201] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1886.697581] Lustre: Skipped 1 previous similar message [ 1886.704586] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1890.486296] Lustre: server umount lustre-MDT0000 complete [ 1891.814782] LustreError: 74368:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1893.667743] LustreError: 74348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788815615 with bad export cookie 4618703408085796325 [ 1893.675020] LustreError: 74348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1893.678137] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1893.938178] Lustre: server umount lustre-MDT0001 complete [ 1903.506480] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [ 1905.652892] Key type lgssc unregistered [ 1905.956597] LNet: 78141:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1905.969379] LNetError: 78141:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1905.998103] LNet: Removed LNI 192.168.202.139@tcp [ 1906.866352] Key type .llcrypt unregistered [ 1906.878106] Key type ._llcrypt unregistered [ 1925.953162] Key type ._llcrypt registered [ 1925.955548] Key type .llcrypt registered [ 1926.016654] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 1926.806646] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1926.856392] alg: No test for adler32 (adler32-zlib) [ 1927.811805] Lustre: Lustre: Build Version: 2.17.58_40_gea5fd3f [ 1927.999685] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [ 1929.704231] Key type lgssc registered [ 1930.604956] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1936.762243] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1936.776466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1938.068499] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1941.539557] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 1947.732054] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1948.056924] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1951.010822] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 1955.928756] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1959.586230] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1963.619505] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1963.920433] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1967.066972] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 1971.449970] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1976.566249] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1977.129865] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1978.622138] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1978.626122] Lustre: Skipped 2 previous similar messages [ 1980.953822] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1980.957861] Lustre: Skipped 3 previous similar messages [ 1985.180975] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1985.183533] Lustre: Skipped 6 previous similar messages [ 1993.568389] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1993.571018] Lustre: Skipped 13 previous similar messages [ 2009.577181] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2009.580243] Lustre: Skipped 28 previous similar messages [ 2040.752235] Lustre: 87612:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2042.973590] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 17:16:03 (1788815763) [ 2044.239059] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2045.769389] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 17:16:06 (1788815766) [ 2048.994846] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2048.996881] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2049.002617] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2049.007250] Lustre: Skipped 1 previous similar message [ 2049.018785] Lustre: Skipped 1 previous similar message [ 2054.122787] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2054.128977] Lustre: Skipped 1 previous similar message [ 2059.238819] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2059.248886] Lustre: Skipped 1 previous similar message [ 2062.304309] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2062.433938] Lustre: server umount lustre-OST0000 complete [ 2065.892625] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2065.902613] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2065.918825] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2069.483616] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2071.387519] Lustre: server umount lustre-MDT0000 complete [ 2073.931526] LustreError: 79511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788815795 with bad export cookie 4487406065755609921 [ 2073.935107] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2073.942601] LustreError: 79511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2074.206307] Lustre: server umount lustre-MDT0001 complete [ 2082.196741] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [ 2084.108984] Key type lgssc unregistered [ 2084.326830] LNet: 88860:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2084.333262] LNetError: 88860:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2085.350670] LNet: Removed LNI 192.168.202.139@tcp [ 2085.953766] Key type .llcrypt unregistered [ 2085.955331] Key type ._llcrypt unregistered [ 2088.780577] Key type ._llcrypt registered [ 2088.781924] Key type .llcrypt registered [ 2088.833211] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2090.894659] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules [ 2091.393552] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2091.416552] alg: No test for adler32 (adler32-zlib) [ 2092.380318] Lustre: Lustre: Build Version: 2.17.58_40_gea5fd3f [ 2092.522147] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [ 2094.152209] Key type lgssc registered [ 2094.830837] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2095.430585] LustreError: 90007:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2106.871150] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 2113.144984] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2116.578971] LustreError: 90007:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2116.604204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2117.956650] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2120.527328] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 2126.302744] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 17:17:27 (1788815847) [ 2127.332167] LustreError: 91039:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2127.344590] LustreError: 91039:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2129.632218] LustreError: 91071:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2129.644774] LustreError: 91071:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2129.657679] LustreError: 91071:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 12, retries 0, failed: rc = -5 [ 2129.849963] Lustre: server umount lustre-MDT0000 complete [ 2139.606878] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [ 2141.547374] Key type lgssc unregistered [ 2141.738488] LNet: 92563:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2141.747399] LNetError: 92563:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2141.756227] LNet: Removed LNI 192.168.202.139@tcp [ 2142.408852] Key type .llcrypt unregistered [ 2142.410906] Key type ._llcrypt unregistered [ 2153.511293] Key type ._llcrypt registered [ 2153.513333] Key type .llcrypt registered [ 2153.587077] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 2154.255087] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2154.310129] alg: No test for adler32 (adler32-zlib) [ 2155.243604] Lustre: Lustre: Build Version: 2.17.58_40_gea5fd3f [ 2155.400297] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [ 2157.032329] Key type lgssc registered [ 2157.890811] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2162.758384] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2162.771343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2164.081980] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2167.248576] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 2172.985465] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2173.526285] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2176.516283] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 2180.980384] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2184.306344] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2187.912491] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2188.287379] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2191.764241] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 2195.715623] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2232.088847] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 17:19:12 (1788815952) [ 2234.504826] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2237.347840] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 17:19:17 (1788815957) [ 2244.579663] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2244.587083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2244.588473] Lustre: Skipped 1 previous similar message [ 2244.597381] Lustre: Skipped 2 previous similar messages [ 2247.387581] Lustre: server umount lustre-MDT0000 complete [ 2249.710691] LustreError: 95684:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2249.728741] LustreError: 95684:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2252.406498] LustreError: 93737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788815974 with bad export cookie 10128479263237938676 [ 2252.409092] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2252.416679] LustreError: 93737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2252.767938] Lustre: server umount lustre-MDT0001 complete [ 2269.756392] Lustre: server umount lustre-OST0000 complete [ 2285.124158] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2285.593646] LustreError: 98027:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2285.837735] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2291.169363] LustreError: 98028:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2291.623952] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 2295.281570] LustreError: 98027:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2300.392397] LustreError: 98028:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2302.685477] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2303.707764] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2309.935884] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 2318.301351] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2324.802130] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2338.179957] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2338.359685] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2338.364346] LustreError: 98015:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2338.386199] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2338.392657] LustreError: 100111:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -98 [ 2338.397713] LustreError: 100111:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 2338.402203] LustreError: 100111:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2338.456755] Lustre: server umount lustre-OST0000 complete [ 2338.459903] LustreError: 100111:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2346.712908] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2346.855493] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2346.863348] LustreError: 98017:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2346.888292] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2346.895994] LustreError: 100575:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -98 [ 2346.902123] LustreError: 100575:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 2346.905587] LustreError: 100575:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2346.992324] Lustre: server umount lustre-OST0000 complete [ 2346.997393] LustreError: 100575:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2355.811359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2355.990678] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2355.995220] Lustre: Found index 0 for lustre-OST0000, updating log [ 2356.006864] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2356.092918] Lustre: lustre-OST0000: new disk, initializing [ 2356.096883] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2356.113389] Lustre: 101112:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2356.539845] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2362.963775] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 2366.346339] Lustre: Failing over lustre-OST0000 [ 2366.490967] Lustre: server umount lustre-OST0000 complete [ 2370.343288] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2370.356532] Lustre: Skipped 1 previous similar message [ 2370.361146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2370.364928] Lustre: Skipped 1 previous similar message [ 2372.460741] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2372.468748] Lustre: Skipped 1 previous similar message [ 2376.723423] Lustre: server umount lustre-MDT0000 complete [ 2377.698708] LustreError: 98028:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2377.721678] LustreError: 98028:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2380.470742] LustreError: 98009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788816102 with bad export cookie 10128479263237941420 [ 2380.476968] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2380.481263] LustreError: 98009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2380.727813] Lustre: server umount lustre-MDT0001 complete [ 2389.427707] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 17:21:50 (1788816110) [ 2407.336360] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 2416.845532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2417.122849] LustreError: 103455:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2417.252161] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2421.472744] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 2429.699410] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2435.329281] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 2438.243277] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2444.813448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2451.604184] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 2459.759476] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2459.831118] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2459.887778] Lustre: lustre-OST0001: new disk, initializing [ 2459.891303] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2459.906225] Lustre: 105774:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2460.155600] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2460.159607] Lustre: Skipped 2 previous similar messages [ 2460.374354] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2460.385180] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2460.477840] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2465.930552] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 2473.198858] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2478.452821] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2482.145936] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2482.163488] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2482.181983] Lustre: Skipped 1 previous similar message [ 2482.198717] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2488.257796] Lustre: server umount lustre-MDT0001 complete [ 2488.295525] LustreError: 103456:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2488.310650] LustreError: 103456:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2497.885322] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2498.019741] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2498.028837] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2498.058562] Lustre: 103446:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2499.467930] LustreError: 107322:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2499.486046] LustreError: 107322:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2499.497176] LustreError: 103448:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2499.647287] LustreError: 107344:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.139@tcp: cfg command failed: rc = -17 [ 2499.661206] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2499.661206] [ 2503.593767] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 2503.695696] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 2514.566833] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 17:23:54 (1788816234) [ 2533.830825] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 2550.223454] Lustre: 109388:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2569.472643] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2572.913155] Lustre: 110522:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2652.130198] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2652.130939] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2652.145246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2652.162555] Lustre: Skipped 5 previous similar messages [ 2652.170760] Lustre: Skipped 7 previous similar messages [ 2655.055976] Lustre: server umount lustre-MDT0000 complete [ 2657.250628] LustreError: 103450:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2657.281932] LustreError: 103450:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [ 2658.794407] LustreError: 103436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788816380 with bad export cookie 10128479263237942414 [ 2658.796514] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2658.806063] LustreError: 103436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2659.157766] Lustre: server umount lustre-MDT0001 complete [ 2673.432380] Lustre: server umount lustre-OST0000 complete [ 2686.248654] Lustre: server umount lustre-OST0001 complete [ 2691.343641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2699.512966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2706.381901] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2714.736381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2731.846872] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 2740.601795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2740.617801] Lustre: lustre-MDT0000: reset Object Index mappings [ 2741.126597] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2741.132996] Lustre: Skipped 1 previous similar message [ 2745.167118] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 2751.916275] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2751.936322] Lustre: lustre-MDT0001: reset Object Index mappings [ 2756.390514] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 2759.095833] Lustre: 115618:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2764.739793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2766.450324] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 2770.342867] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 2776.729210] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2777.070601] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 2781.476022] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 2782.715794] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 2782.734300] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 2788.016531] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2791.383576] Lustre: 117495:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2834.833172] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 17:29:15 (1788816555) [ 2864.615594] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2864.619960] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2864.633740] Lustre: Skipped 3 previous similar messages [ 2864.639090] Lustre: Skipped 2 previous similar messages [ 2874.337036] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2874.589269] Lustre: server umount lustre-MDT0000 complete [ 2874.857694] LustreError: 114467:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2874.869690] LustreError: 114467:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [ 2877.258056] LustreError: 114452:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788816599 with bad export cookie 10128479263237989265 [ 2877.259688] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2877.268785] LustreError: 114452:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2879.978333] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2882.984539] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2882.990628] Lustre: Skipped 8 previous similar messages [ 2883.788500] Lustre: server umount lustre-MDT0001 complete [ 2892.932711] Lustre: server umount lustre-OST0000 complete [ 2901.966827] Lustre: server umount lustre-OST0001 complete [ 2906.675981] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_hostid [ 2913.308830] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 2939.300770] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2939.506450] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2939.527536] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2939.584354] Lustre: lustre-MDT0000: new disk, initializing [ 2939.646730] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2939.655597] Lustre: Skipped 3 previous similar messages [ 2939.666970] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2942.915362] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 2948.033759] Lustre: server umount lustre-MDT0000 complete [ 2955.048152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2955.228525] Lustre: Found index 0 for lustre-MDT0000, updating log [ 2955.238732] Lustre: 121765:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2955.253159] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 2958.525415] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 2962.040575] Lustre: server umount lustre-MDT0000 complete [ 2968.396285] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 17:31:29 (1788816689) [ 2969.570040] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 2971.045911] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 17:31:31 (1788816691) [ 2983.751621] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 2991.581616] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2995.642203] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 3002.275534] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3002.361376] Lustre: 123816:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3002.412527] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3002.415357] Lustre: Skipped 1 previous similar message [ 3002.464142] Lustre: lustre-MDT0001: new disk, initializing [ 3002.607198] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3002.621487] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3006.519198] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 3010.280718] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3015.647163] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3015.936677] Lustre: lustre-OST0000: new disk, initializing [ 3015.940307] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3015.945867] Lustre: 125392:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3017.076707] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3017.084264] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3017.203051] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3021.275537] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 3029.603473] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3029.701607] Lustre: lustre-OST0001: new disk, initializing [ 3029.704389] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3029.709568] Lustre: 126264:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3031.218958] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3031.228932] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3031.324696] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3034.456300] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 3041.993722] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3044.502182] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3315.263157] Lustre: 132660:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3315.270037] Lustre: 132660:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3315.274434] Lustre: 132660:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3315.278562] Lustre: 132660:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3315.282745] Lustre: 132660:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3315.285057] Lustre: 132660:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3518.970940] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 17:40:39 (1788817239) [ 3532.581648] Lustre: server umount lustre-MDT0000 complete [ 3535.165698] LustreError: 132692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788817257 with bad export cookie 10128479263238054414 [ 3535.178204] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3536.353147] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3536.355029] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3536.362705] LustreError: 127049:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3536.362718] LustreError: 127049:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 3536.363970] Lustre: Skipped 4 previous similar messages [ 3536.403812] Lustre: Skipped 1 previous similar message [ 3540.386850] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3540.398892] Lustre: Skipped 1 previous similar message [ 3541.597327] Lustre: server umount lustre-MDT0001 complete [ 3550.875677] Lustre: server umount lustre-OST0000 complete [ 3559.762684] Lustre: server umount lustre-OST0001 complete [ 3564.243954] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_hostid [ 3569.880906] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 3593.345686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3593.554896] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3593.583922] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3593.676376] Lustre: lustre-MDT0000: new disk, initializing [ 3593.735577] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3593.742265] Lustre: Skipped 5 previous similar messages [ 3593.752365] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3596.298099] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 3606.030318] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3606.122212] Lustre: 135697:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3606.311534] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3612.402672] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 3626.419487] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3636.406835] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3645.053540] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3645.318461] Lustre: lustre-OST0000: new disk, initializing [ 3645.320823] Lustre: Skipped 1 previous similar message [ 3645.347356] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3645.354568] Lustre: Skipped 2 previous similar messages [ 3645.361687] Lustre: 137668:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3646.662477] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3646.685085] Lustre: Skipped 1 previous similar message [ 3646.699041] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3646.959504] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3653.317677] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 3665.201617] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3677.525708] Lustre: lustre-MDD0000: changelog on [ 3680.468611] Lustre: lustre-MDD0001: changelog on [ 5404.801833] Lustre: lustre-MDD0001: changelog off [ 5406.632572] Lustre: 139285:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 5407.472817] Lustre: lustre-MDD0000: changelog off [ 5413.347266] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5413.354954] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5413.365427] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5413.858460] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5413.869707] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5416.436860] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5416.441347] Lustre: Skipped 2 previous similar messages [ 5418.947478] Lustre: server umount lustre-OST0000 complete [ 5424.103174] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5424.113399] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5424.130804] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5426.659352] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5429.866878] Lustre: server umount lustre-MDT0000 complete [ 5431.782366] LustreError: 135704:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5431.806776] LustreError: 135704:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 5434.212255] LustreError: 135690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788819156 with bad export cookie 10128479263238868388 [ 5434.228444] LustreError: 135690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 5434.231258] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5434.658677] Lustre: server umount lustre-MDT0001 complete [ 5447.522571] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [ 5450.104469] Key type lgssc unregistered [ 5450.343865] LNet: 146793:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5450.353552] LNetError: 146793:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5450.394717] LNet: Removed LNI 192.168.202.139@tcp [ 5451.168350] Key type .llcrypt unregistered [ 5451.176337] Key type ._llcrypt unregistered [ 5470.348535] Key type ._llcrypt registered [ 5470.351411] Key type .llcrypt registered [ 5470.456706] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_hostid [ 5483.588113] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 5484.393113] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5484.410783] alg: No test for adler32 (adler32-zlib) [ 5485.503248] Lustre: Lustre: Build Version: 2.17.58_40_gea5fd3f [ 5485.786125] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [ 5487.472182] Key type lgssc registered [ 5488.716118] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5527.770248] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5527.844452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5529.087644] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5529.111261] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5529.216255] Lustre: lustre-MDT0000: new disk, initializing [ 5529.256232] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5529.263585] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5534.451636] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 5548.563922] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5548.809849] Lustre: 149983:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5548.910390] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5548.915277] Lustre: Skipped 1 previous similar message [ 5549.039825] Lustre: lustre-MDT0001: new disk, initializing [ 5549.132113] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5549.166845] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5549.182770] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5554.473882] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 5565.424626] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5571.171705] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5578.363796] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5578.607396] Lustre: lustre-OST0000: new disk, initializing [ 5578.611142] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5578.615486] Lustre: 151955:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5578.749815] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5584.745759] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 5586.458402] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5586.472084] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5586.587390] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5594.037561] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5599.102934] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_module obdecho/obdecho [ 5600.802830] LustreError: 152959:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 5600.813960] LustreError: 152959:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 5602.485482] LustreError: 153055:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 5615.585972] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5615.594665] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5615.604995] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5617.125987] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5617.133486] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5617.148384] Lustre: Skipped 1 previous similar message [ 5617.171263] Lustre: Skipped 2 previous similar messages [ 5619.187603] Lustre: server umount lustre-MDT0000 complete [ 5622.431805] LustreError: 149976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788819344 with bad export cookie 3427404182104257787 [ 5622.438664] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5622.608614] Lustre: server umount lustre-MDT0001 complete [ 5636.339488] Lustre: server umount lustre-OST0000 complete [ 5655.976988] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_hostid [ 5669.565940] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 5714.164508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5714.497598] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5714.561911] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5714.620598] Lustre: lustre-MDT0000: new disk, initializing [ 5714.717843] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5714.732666] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5720.163483] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 5733.318914] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5733.436863] Lustre: 156771:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5733.550704] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5733.556684] Lustre: Skipped 1 previous similar message [ 5733.810318] Lustre: lustre-MDT0001: new disk, initializing [ 5734.034933] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5734.081095] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5734.097716] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5741.501620] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 5754.766518] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5763.655081] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5773.290341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5773.642649] Lustre: lustre-OST0000: new disk, initializing [ 5773.646855] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5773.652617] Lustre: 158739:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5773.875144] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5782.365950] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 5783.153762] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5783.178048] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5783.361465] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5796.778295] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5805.388216] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5805.754583] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5811.918790] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5812.229920] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5819.362836] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5819.374132] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5819.379182] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5819.391758] Lustre: Skipped 1 previous similar message [ 5819.405313] Lustre: Skipped 1 previous similar message [ 5820.999777] Lustre: server umount lustre-OST0000 complete [ 5826.017627] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5826.033315] LustreError: Skipped 1 previous similar message [ 5826.049317] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5826.076740] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5831.914921] Lustre: server umount lustre-MDT0000 complete [ 5834.727070] LustreError: 156779:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5834.758729] LustreError: 156779:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 5835.986986] LustreError: 156764:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788819557 with bad export cookie 3427404182104258746 [ 5835.991383] LustreError: 156764:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5835.996852] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5836.449652] Lustre: server umount lustre-MDT0001 complete [ 5840.656324] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 18:19:20 (1788819560) [ 5842.976736] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 5845.685541] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 18:19:25 (1788819565) [ 5855.077633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5855.488663] LustreError: 161158:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5855.593116] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5860.790168] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 5860.841892] LustreError: 161159:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5865.982914] LustreError: 161158:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5871.075925] LustreError: 161159:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5871.246236] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5878.078514] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 5887.189405] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5893.156545] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5900.578994] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5900.714238] LustreError: 162978:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5900.995497] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5900.999709] Lustre: Skipped 1 previous similar message [ 5907.653924] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 5917.204407] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5923.756894] LustreError: 163737:0:(ldlm_resource.c:1207:ldlm_resource_complain()) mdt-lustre-MDT0000_UUID: namespace resource [0x200000bd0:0x1:0x0].0x14119535 (000000009b65ae31) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 5926.883334] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5926.886200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5926.890445] Lustre: Skipped 2 previous similar messages [ 5926.895279] Lustre: Skipped 4 previous similar messages [ 5935.017189] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5935.031051] Lustre: Skipped 7 previous similar messages [ 5938.144268] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5938.286838] Lustre: server umount lustre-MDT0000 complete [ 5938.602105] LustreError: 161155:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5942.385407] LustreError: 161140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788819664 with bad export cookie 3427404182104259628 [ 5942.414900] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5942.422086] LustreError: 161140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 5945.314907] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5945.338390] Lustre: Skipped 1 previous similar message [ 5948.928371] Lustre: server umount lustre-MDT0001 complete [ 5956.353291] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 5969.212159] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5969.447261] LustreError: 165001:0:(llog.c:1655:llog_backup()) MGC192.168.202.139@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 5969.461199] Lustre: 165001:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.139@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 5972.961603] LustreError: 165006:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5972.973977] LustreError: 165006:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 6030.305279] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6030.317740] Lustre: 165001:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.139@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6038.052891] LustreError: 162990:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6038.086613] LustreError: 162990:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 49 previous similar messages [ 6071.283983] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6076.191708] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 6076.405398] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6084.227816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6088.682644] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x2f9095f1c8035562 [ 6088.691976] Lustre: MGC192.168.202.139@tcp: Connection restored to 0@lo (at 0@lo) [ 6093.739581] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 6094.311087] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6100.593294] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6102.013828] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6107.411850] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6108.839860] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6125.026170] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6125.030841] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6125.049089] Lustre: Skipped 1 previous similar message [ 6125.057258] Lustre: Skipped 3 previous similar messages [ 6126.405916] Lustre: server umount lustre-MDT0000 complete [ 6129.700286] LustreError: 161139:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788819851 with bad export cookie 3427404182104266082 [ 6129.706437] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6129.710736] LustreError: 161139:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6130.114286] Lustre: server umount lustre-MDT0001 complete [ 6144.129809] Lustre: server umount lustre-OST0000 complete [ 6169.019902] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_hostid [ 6185.336475] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 6225.299353] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6225.565781] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6225.613460] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6225.746277] Lustre: lustre-MDT0000: new disk, initializing [ 6225.854643] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6225.866195] Lustre: Skipped 1 previous similar message [ 6225.898250] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6231.405742] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 6242.393550] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6242.521488] Lustre: 170129:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6242.792512] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6247.784502] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 6258.142530] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6264.419648] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6271.718565] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6272.008618] Lustre: lustre-OST0000: new disk, initializing [ 6272.012739] Lustre: Skipped 1 previous similar message [ 6272.016083] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6272.021657] Lustre: Skipped 2 previous similar messages [ 6272.025880] Lustre: 172100:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6279.802809] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 6280.228930] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6280.238209] Lustre: Skipped 1 previous similar message [ 6280.245504] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6280.375782] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6291.443363] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6297.830210] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6298.097914] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6303.108067] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6303.360573] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6306.279450] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6306.284308] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6306.288396] Lustre: Skipped 1 previous similar message [ 6306.290508] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6306.290519] Lustre: Skipped 2 previous similar messages [ 6306.342841] LustreError: Skipped 1 previous similar message [ 6311.259621] Lustre: server umount lustre-OST0000 complete [ 6321.645106] LustreError: 170142:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6321.657563] LustreError: 170142:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 38 previous similar messages [ 6321.727852] Lustre: server umount lustre-MDT0000 complete [ 6325.909564] LustreError: 170927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788820047 with bad export cookie 3427404182104282917 [ 6325.916568] LustreError: 170927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6325.916727] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6326.413935] Lustre: server umount lustre-MDT0001 complete [ 6335.978228] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 18:27:36 (1788820056) [ 6344.023617] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6349.074965] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 6358.950743] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6366.252170] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 6375.997996] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6381.418584] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6387.902801] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6396.120793] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 6404.575222] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6414.394576] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 18:28:55 (1788820135) [ 6419.964538] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6419.967075] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6419.977388] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6419.977400] Lustre: Skipped 3 previous similar messages [ 6419.978697] LustreError: Skipped 1 previous similar message [ 6420.022335] Lustre: Skipped 4 previous similar messages [ 6425.686962] Lustre: server umount lustre-OST0000 complete [ 6431.205168] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6437.064468] Lustre: server umount lustre-MDT0000 complete [ 6440.789181] LustreError: 174491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788820162 with bad export cookie 3427404182104283799 [ 6440.800087] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6440.806954] LustreError: 174491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6441.094695] Lustre: server umount lustre-MDT0001 complete [ 6453.796911] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [ 6457.708563] Key type lgssc unregistered [ 6458.055900] LNet: 178238:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6458.062645] LNetError: 178238:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6458.078165] LNet: Removed LNI 192.168.202.139@tcp [ 6458.984349] Key type .llcrypt unregistered [ 6458.986134] Key type ._llcrypt unregistered [ 6480.334601] Key type ._llcrypt registered [ 6480.340919] Key type .llcrypt registered [ 6480.431517] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 6481.812176] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6481.827485] alg: No test for adler32 (adler32-zlib) [ 6482.853458] Lustre: Lustre: Build Version: 2.17.58_40_gea5fd3f [ 6483.122617] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [ 6484.928310] Key type lgssc registered [ 6486.149939] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6497.307843] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6497.340428] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6512.866983] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6517.984796] LustreError: 179479:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.139@tcp: failed processing log, type 4: rc = -110 [ 6523.168585] LustreError: 179479:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 6523.193886] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 6523.219762] LustreError: MGC192.168.202.139@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 6523.243810] LustreError: 179479:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-OST0000: -22 [ 6523.258210] LustreError: 179479:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -22 [ 6523.268861] LustreError: 179479:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 6523.357437] Lustre: server umount lustre-OST0000 complete [ 6523.359844] LustreError: 179479:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6532.745150] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6534.324260] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6538.431388] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 6544.874827] LustreError: 179978:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6544.900513] LustreError: 179978:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6546.944305] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6547.369396] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6552.102090] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 6559.276757] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6564.689629] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6571.665922] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6571.857909] LustreError: 181793:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 6571.867760] Lustre: 181793:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.202.139@tcp: invalid local config log lustre-OST0000: rc = -22 [ 6572.012689] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6579.628490] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 6587.198627] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6589.411559] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6589.422231] Lustre: Skipped 1 previous similar message [ 6589.423640] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6589.438163] Lustre: Skipped 2 previous similar messages [ 6594.539244] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6595.843551] Lustre: server umount lustre-MDT0000 complete [ 6599.394230] LustreError: 179959:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788820321 with bad export cookie 664898497486776831 [ 6599.414845] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6599.651938] LustreError: 179973:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6599.664680] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6600.049666] Lustre: server umount lustre-MDT0001 complete [ 6606.791652] Lustre: server umount lustre-OST0000 complete [ 6616.858792] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_hostid [ 6627.160829] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 6666.026396] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6666.279328] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6666.300414] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6666.391794] Lustre: lustre-MDT0000: new disk, initializing [ 6666.463436] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6666.475596] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6670.362926] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 6680.716255] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6680.817438] Lustre: 185316:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6680.861917] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6680.865453] Lustre: Skipped 1 previous similar message [ 6680.918217] Lustre: lustre-MDT0001: new disk, initializing [ 6680.991038] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6681.024711] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6681.034356] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6685.533908] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 6695.800265] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6701.629119] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6708.118744] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6708.343577] Lustre: lustre-OST0000: new disk, initializing [ 6708.346910] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6708.352384] Lustre: 187287:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6708.427543] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6714.474509] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6714.488993] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6714.692125] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6715.201818] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 6725.991642] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6733.050307] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6733.307321] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6738.512626] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6738.728230] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6745.591137] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6745.592948] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6745.609570] LustreError: Skipped 1 previous similar message [ 6745.612808] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6745.626730] Lustre: Skipped 1 previous similar message [ 6745.651240] Lustre: Skipped 3 previous similar messages [ 6747.033229] Lustre: server umount lustre-OST0000 complete [ 6752.740404] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6752.752593] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6752.778799] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6760.931359] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6760.938710] Lustre: Skipped 3 previous similar messages [ 6766.048162] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6766.325322] Lustre: server umount lustre-MDT0000 complete [ 6770.167344] LustreError: 185328:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6770.177378] LustreError: 185328:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6770.367360] LustreError: 185308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788820492 with bad export cookie 664898497486777552 [ 6770.381517] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6770.391910] LustreError: 185308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 6770.786649] Lustre: server umount lustre-MDT0001 complete [ 6780.545536] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 18:35:00 (1788820500) [ 6801.081444] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [ 6804.013426] Key type lgssc unregistered [ 6804.236884] LNet: 190233:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6804.241119] LNetError: 190233:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6804.251370] LNet: Removed LNI 192.168.202.139@tcp [ 6804.890252] Key type .llcrypt unregistered [ 6804.891882] Key type ._llcrypt unregistered [ 6825.261116] Key type ._llcrypt registered [ 6825.263683] Key type .llcrypt registered [ 6825.405911] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 6826.573176] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6826.616342] alg: No test for adler32 (adler32-zlib) [ 6827.795637] Lustre: Lustre: Build Version: 2.17.58_40_gea5fd3f [ 6828.032462] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [ 6829.760288] Key type lgssc registered [ 6831.346284] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6841.386358] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6841.434646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6843.169228] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6849.963886] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 6853.619102] LustreError: 191450:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6853.658775] LustreError: 191450:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6858.728959] LustreError: 191449:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6859.261915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6859.537551] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6863.545177] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 6870.761554] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6876.596061] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6884.445973] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6884.575134] Lustre: *** cfs_fail_loc=131e, val=0*** [ 6884.576885] LustreError: 193315:0:(llog.c:1696:llog_backup()) MGC192.168.202.139@tcp: failed to backup log lustre-OST0000: rc = -28 [ 6884.582265] Lustre: 193315:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.202.139@tcp: can't backup local config lustre-OST0000: rc = -28 [ 6884.701683] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6892.715505] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 6901.309508] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6904.289914] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6904.294520] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6904.308568] Lustre: Skipped 1 previous similar message [ 6905.378604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6905.394899] Lustre: Skipped 4 previous similar messages [ 6909.955055] Lustre: server umount lustre-MDT0000 complete [ 6910.441503] LustreError: 191446:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6910.471099] LustreError: 191446:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6913.202276] LustreError: 191430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788820635 with bad export cookie 15931040003043779053 [ 6913.209837] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6913.223390] LustreError: 191430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6913.592349] LustreError: 192180:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000a36f902f x1875714297065088/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6913.904903] Lustre: server umount lustre-MDT0001 complete [ 6928.558929] Lustre: server umount lustre-OST0000 complete [ 6939.012607] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_hostid [ 6951.807633] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 6985.534889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6985.825616] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6985.851911] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6985.911789] Lustre: lustre-MDT0000: new disk, initializing [ 6985.971487] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6985.985571] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6990.741216] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 7000.631180] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7000.737894] Lustre: 196841:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7000.797948] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7000.801349] Lustre: Skipped 1 previous similar message [ 7000.807757] LustreError: 197623:0:(llog.c:1696:llog_backup()) MGC192.168.202.139@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7000.816574] LustreError: 197623:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7000.820697] Lustre: 197623:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.139@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7000.831875] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7000.837645] Lustre: Skipped 1 previous similar message [ 7000.933840] Lustre: lustre-MDT0001: new disk, initializing [ 7001.050089] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7001.084724] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7001.092455] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7005.239778] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 7013.738897] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7018.369015] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7023.406347] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7023.623680] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7023.625671] Lustre: Skipped 1 previous similar message [ 7023.627371] LustreError: 198785:0:(llog.c:1696:llog_backup()) MGC192.168.202.139@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7023.632168] LustreError: 198785:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7023.637619] Lustre: 198785:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.139@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7023.644258] Lustre: 198785:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7023.662981] Lustre: lustre-OST0000: new disk, initializing [ 7023.666597] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7023.671323] Lustre: 198808:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7023.755247] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7029.727686] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 7032.417201] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7032.433834] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7032.553247] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7039.632795] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7045.416746] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7045.592416] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7050.032539] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7050.216462] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7053.281170] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7053.283597] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7053.294811] LustreError: Skipped 1 previous similar message [ 7053.295604] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7053.319605] Lustre: Skipped 2 previous similar messages [ 7058.391601] Lustre: server umount lustre-OST0000 complete [ 7062.498370] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7062.509765] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7062.547764] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7062.550570] Lustre: Skipped 1 previous similar message [ 7067.877547] Lustre: server umount lustre-MDT0000 complete [ 7068.642146] LustreError: 196852:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7068.656486] LustreError: 196852:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7071.441247] LustreError: 196832:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788820793 with bad export cookie 15931040003043779774 [ 7071.460113] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7071.464191] LustreError: 196832:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7071.887587] Lustre: server umount lustre-MDT0001 complete [ 7080.101902] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 18:40:00 (1788820800) [ 7094.816547] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 7104.113539] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7104.390175] LustreError: 201933:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7104.435378] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7108.771734] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 7114.722822] LustreError: 201934:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7114.741953] LustreError: 201934:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7115.498267] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7120.087698] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 7122.722376] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7128.507039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7135.667972] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 7145.075046] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7145.310633] Lustre: lustre-OST0001: new disk, initializing [ 7145.317467] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7145.326162] Lustre: 204240:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7145.440463] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7145.448413] Lustre: Skipped 2 previous similar messages [ 7152.353291] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7152.373891] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7152.519980] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7153.235285] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 7163.850202] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7169.039815] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7170.704933] Lustre: DEBUG MARKER: ADD OST3 [ 7175.134637] loop: module loaded [ 7175.199747] loop0: detected capacity change from 0 to 204800000 [ 7175.453349] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7175.466960] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7175.486280] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7180.661591] loop0: detected capacity change from 0 to 204800000 [ 7182.309276] Lustre: DEBUG MARKER: STOP OST3 [ 7186.524215] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7186.698721] Lustre: lustre-OST0003: new disk, initializing [ 7186.701447] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 7186.711427] Lustre: 206195:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7188.151379] Lustre: *** cfs_fail_loc=2109, val=2*** [ 7188.313903] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 7188.324464] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 7188.358948] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 7190.176813] LustreError: 206374:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 7190.995847] Lustre: Failing over lustre-OST0003 [ 7191.170892] Lustre: server umount lustre-OST0003 complete [ 7192.826666] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 7193.576108] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7193.601867] Lustre: Skipped 1 previous similar message [ 7193.611217] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7193.612315] LustreError: 203421:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7193.636602] LustreError: 203421:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7202.318506] Lustre: DEBUG MARKER: START OST3 again [ 7208.517276] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7208.665240] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 7210.351349] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 7210.553560] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 7210.559604] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7213.603697] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 7231.986948] Lustre: Failing over lustre-OST0003 [ 7232.085529] Lustre: server umount lustre-OST0003 complete [ 7234.568845] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7234.589988] Lustre: Skipped 1 previous similar message [ 7234.598336] LustreError: 203420:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7234.620039] LustreError: 203420:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [ 7249.895579] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7249.898209] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7249.902126] Lustre: Skipped 3 previous similar messages [ 7249.909787] Lustre: Skipped 3 previous similar messages [ 7260.134853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7260.148156] Lustre: Skipped 7 previous similar messages [ 7264.232680] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7264.736627] Lustre: server umount lustre-MDT0000 complete [ 7270.105552] LustreError: 201913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788820992 with bad export cookie 15931040003043780656 [ 7270.112088] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7270.121275] LustreError: 201913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7270.380871] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7270.410197] Lustre: Skipped 2 previous similar messages [ 7277.061471] Lustre: server umount lustre-MDT0001 complete [ 7282.149600] Lustre: server umount lustre-OST0000 complete [ 7285.644900] Lustre: server umount lustre-OST0001 complete [ 7291.543691] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_hostid [ 7299.197848] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 7346.230372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7346.559569] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7346.672609] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7346.810924] Lustre: lustre-MDT0000: new disk, initializing [ 7346.876359] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7346.885602] Lustre: Skipped 2 previous similar messages [ 7346.896974] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7351.276878] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 7361.754096] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7361.979220] Lustre: 211702:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7362.347645] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7368.501861] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 7378.006164] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7385.070371] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7392.077699] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7392.538257] Lustre: lustre-OST0000: new disk, initializing [ 7392.540800] Lustre: Skipped 1 previous similar message [ 7392.547452] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7392.551742] Lustre: Skipped 2 previous similar messages [ 7392.558268] Lustre: 213669:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7394.594924] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7394.611159] Lustre: Skipped 1 previous similar message [ 7394.613048] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7394.890927] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7400.495888] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 7410.616905] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7415.498788] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7415.695204] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7420.115158] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7420.455562] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7425.512482] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7425.516255] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7425.519194] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7425.525063] Lustre: Skipped 1 previous similar message [ 7425.536578] Lustre: Skipped 5 previous similar messages [ 7428.651544] Lustre: server umount lustre-OST0000 complete [ 7434.213298] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7434.231893] LustreError: Skipped 1 previous similar message [ 7439.655075] Lustre: server umount lustre-MDT0000 complete [ 7440.873439] LustreError: 211714:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7440.923158] LustreError: 211714:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 17 previous similar messages [ 7445.099304] LustreError: 211693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788821167 with bad export cookie 15931040003043782945 [ 7445.104094] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7445.118635] LustreError: 211693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7445.555628] Lustre: server umount lustre-MDT0001 complete [ 7451.748974] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 18:46:11 (1788821171) [ 7472.305698] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_hostid [ 7482.767294] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 7526.787460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7527.130378] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7527.179913] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7527.257781] Lustre: lustre-MDT0000: new disk, initializing [ 7527.324409] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7527.330483] Lustre: Skipped 2 previous similar messages [ 7527.342958] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7532.327727] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 7543.818544] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7544.086705] Lustre: 218283:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7544.471079] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7549.647337] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 7560.018295] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7565.132936] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7572.324060] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7572.566154] Lustre: 220248:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7574.622976] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7579.818602] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 7590.006507] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7595.161508] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7595.497607] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7601.238901] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7601.469380] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7605.227556] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7605.232609] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7605.237194] LustreError: Skipped 1 previous similar message [ 7605.239045] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7605.261746] Lustre: Skipped 3 previous similar messages [ 7605.264690] Lustre: Skipped 3 previous similar messages [ 7610.440075] Lustre: server umount lustre-OST0000 complete [ 7630.304197] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7630.803774] Lustre: server umount lustre-MDT0000 complete [ 7630.838455] LustreError: 219142:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7630.853438] LustreError: 219142:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7635.590318] LustreError: 218274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788821357 with bad export cookie 15931040003043783834 [ 7635.616765] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7635.618528] LustreError: 218274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7636.004728] Lustre: server umount lustre-MDT0001 complete [ 7646.049032] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7651.729409] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 7661.516809] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7668.487750] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 7679.112880] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7686.686710] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7693.739090] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7701.204674] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 7709.422478] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7727.078389] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7727.090649] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7727.095056] Lustre: Skipped 7 previous similar messages [ 7727.113745] LustreError: Skipped 1 previous similar message [ 7731.514339] Lustre: server umount lustre-OST0000 complete [ 7737.319144] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7737.335024] Lustre: Skipped 4 previous similar messages [ 7743.376500] Lustre: server umount lustre-MDT0000 complete [ 7746.918066] LustreError: 222449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788821468 with bad export cookie 15931040003043784716 [ 7746.925061] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7746.928986] LustreError: 222449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7747.540656] Lustre: server umount lustre-MDT0001 complete [ 7848.740600] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [ 7851.375763] Key type lgssc unregistered [ 7851.743951] LNet: 226155:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7851.772340] LNetError: 226155:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7852.838297] LNet: Removed LNI 192.168.202.139@tcp [ 7853.746156] Key type .llcrypt unregistered [ 7853.748888] Key type ._llcrypt unregistered [ 7878.343454] Key type ._llcrypt registered [ 7878.350251] Key type .llcrypt registered [ 7878.520457] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_hostid [ 7894.810101] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 7895.944569] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7895.971155] alg: No test for adler32 (adler32-zlib) [ 7897.246934] Lustre: Lustre: Build Version: 2.17.58_40_gea5fd3f [ 7897.720463] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [ 7899.424212] Key type lgssc registered [ 7901.297298] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7943.865961] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7943.893948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7945.076587] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7945.094641] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7945.151519] Lustre: lustre-MDT0000: new disk, initializing [ 7945.207698] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7945.221550] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7949.517484] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 7960.105981] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7960.241561] Lustre: 229509:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7960.289118] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7960.292542] Lustre: Skipped 1 previous similar message [ 7960.376462] Lustre: lustre-MDT0001: new disk, initializing [ 7960.476863] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7960.507524] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7960.531906] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7965.145708] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 7976.947901] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7983.987108] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7989.878094] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7990.050438] Lustre: lustre-OST0000: new disk, initializing [ 7990.058637] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7990.063889] Lustre: 231480:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7990.163892] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7996.638837] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 7997.017591] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7997.029616] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7997.161686] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8005.873526] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8010.277308] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8010.436362] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8015.129918] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8015.308975] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8017.891001] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8017.906458] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8017.910028] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8017.917513] Lustre: Skipped 1 previous similar message [ 8017.934368] Lustre: Skipped 1 previous similar message [ 8023.015225] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8023.021564] Lustre: Skipped 1 previous similar message [ 8028.131222] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8028.138186] Lustre: Skipped 1 previous similar message [ 8031.200955] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8031.475985] Lustre: server umount lustre-OST0000 complete [ 8037.349824] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8037.357206] LustreError: Skipped 1 previous similar message [ 8037.361167] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8037.376986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8038.371072] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8041.497397] Lustre: server umount lustre-MDT0000 complete [ 8044.835716] LustreError: 229502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788821766 with bad export cookie 16652396370746489350 [ 8044.837018] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8044.852559] LustreError: 229502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8045.638940] Lustre: server umount lustre-MDT0001 complete [ 8064.802479] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 8074.839680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8075.121078] LustreError: 234415:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8075.129852] LustreError: 234415:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 8075.159157] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8079.035441] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 8080.359598] LustreError: 234416:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8085.477830] LustreError: 234415:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8086.274935] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8086.539045] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8090.847920] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 8093.873285] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8100.029916] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8100.491770] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8107.283399] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 8117.079854] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8117.374280] Lustre: lustre-OST0001: new disk, initializing [ 8117.383749] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8117.390157] Lustre: 236721:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8117.560080] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8124.001484] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 8125.087398] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8125.102717] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8125.301944] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8135.499907] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8141.165247] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8155.630473] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 18:57:55 (1788821875) [ 8161.259205] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8161.271884] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8161.293504] Lustre: Skipped 2 previous similar messages [ 8161.314762] Lustre: Skipped 5 previous similar messages [ 8164.694681] Lustre: server umount lustre-MDT0000 complete [ 8166.376208] LustreError: 235143:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8166.399653] LustreError: 235143:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 8168.224329] LustreError: 236720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788821890 with bad export cookie 16652396370746490239 [ 8168.230699] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8168.232787] LustreError: 236720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8168.509190] Lustre: server umount lustre-MDT0001 complete [ 8181.262163] Lustre: server umount lustre-OST0000 complete [ 8195.158447] Lustre: server umount lustre-OST0001 complete [ 8200.798661] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_hostid [ 8212.985163] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 8253.088413] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8253.305366] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8253.333768] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8253.410384] Lustre: lustre-MDT0000: new disk, initializing [ 8253.482629] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8253.499793] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8257.328937] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 8267.347130] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8267.497750] Lustre: 240671:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8267.556835] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8267.561777] Lustre: Skipped 1 previous similar message [ 8267.663460] Lustre: lustre-MDT0001: new disk, initializing [ 8267.791472] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8267.808629] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8272.439488] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 8282.897184] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8288.692818] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8296.814645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8297.068562] Lustre: lustre-OST0000: new disk, initializing [ 8297.071818] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8297.076499] Lustre: 242637:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8298.771987] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8298.791425] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8298.941769] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8305.340509] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 8317.926692] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8324.708545] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8324.980676] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8331.029430] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8331.574399] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8334.817989] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8334.823964] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8334.847902] Lustre: Skipped 1 previous similar message [ 8334.865833] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8348.128141] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8348.333370] Lustre: server umount lustre-OST0000 complete [ 8354.787106] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8354.802447] LustreError: Skipped 1 previous similar message [ 8354.814339] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8354.860959] Lustre: Skipped 1 previous similar message [ 8354.890000] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8354.899872] Lustre: Skipped 5 previous similar messages [ 8366.048183] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8366.458556] Lustre: server umount lustre-MDT0000 complete [ 8369.542409] LustreError: 240662:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788822091 with bad export cookie 16652396370746492374 [ 8369.550932] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8369.560107] LustreError: 240662:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8369.942526] Lustre: server umount lustre-MDT0001 complete [ 8377.534555] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8377.898514] LustreError: 244869:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8377.958283] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8377.967738] Lustre: Skipped 2 previous similar messages [ 8382.075713] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 8387.555398] LustreError: 244870:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8387.588911] LustreError: 244870:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8390.586615] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8395.812933] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 8403.189292] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8409.048344] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8415.995480] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8422.163321] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 8429.348281] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8574.435866] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8574.450697] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8574.467237] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8574.469412] Lustre: Skipped 2 previous similar messages [ 8574.488811] Lustre: Skipped 7 previous similar messages [ 8579.875648] Lustre: server umount lustre-OST0000 complete [ 8589.793395] LustreError: 244869:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8589.809779] LustreError: 244869:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8589.830055] Lustre: server umount lustre-MDT0000 complete [ 8593.680937] LustreError: 244849:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788822315 with bad export cookie 16652396370746493256 [ 8593.682914] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8593.692725] LustreError: 244849:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 8594.072913] Lustre: server umount lustre-MDT0001 complete [ 8698.030446] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [ 8700.350805] Key type lgssc unregistered [ 8700.586161] LNet: 248477:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8700.592405] LNetError: 248477:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8700.616247] LNet: Removed LNI 192.168.202.139@tcp [ 8701.511820] Key type .llcrypt unregistered [ 8701.514120] Key type ._llcrypt unregistered [ 8726.096485] Key type ._llcrypt registered [ 8726.099527] Key type .llcrypt registered [ 8726.209328] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_hostid [ 8742.860325] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 8744.751780] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8744.877701] alg: No test for adler32 (adler32-zlib) [ 8746.024853] Lustre: Lustre: Build Version: 2.17.58_40_gea5fd3f [ 8746.398565] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [ 8748.064230] Key type lgssc registered [ 8749.444044] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8802.755678] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 8813.249892] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8813.295045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8814.628308] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8814.655335] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8814.782183] Lustre: lustre-MDT0000: new disk, initializing [ 8814.846661] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8814.863752] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8819.954358] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 8830.049317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8830.165163] Lustre: 252545:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8830.210676] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8830.215773] Lustre: Skipped 1 previous similar message [ 8830.276223] Lustre: lustre-MDT0001: new disk, initializing [ 8830.366899] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8830.407215] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8830.422161] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8835.699084] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 8841.105825] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8849.298302] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8850.018419] Lustre: lustre-OST0000: new disk, initializing [ 8850.026212] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8850.047879] Lustre: 254177:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8850.241488] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8851.801162] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8851.815416] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8851.972394] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8857.562684] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 8868.301952] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8868.468214] Lustre: lustre-OST0001: new disk, initializing [ 8868.471677] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8868.479404] Lustre: 255047:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8868.614744] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8874.639941] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8874.654869] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8874.753200] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8875.356302] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 8887.903466] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8894.760774] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8907.234377] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8907.237576] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8907.275831] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8910.826253] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8910.829816] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8910.840628] Lustre: Skipped 1 previous similar message [ 8910.861641] Lustre: Skipped 3 previous similar messages [ 8912.832885] Lustre: server umount lustre-MDT0000 complete [ 8916.357940] LustreError: 252535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788822638 with bad export cookie 15810612694116727383 [ 8916.361552] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8916.370385] LustreError: 252535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8916.781543] Lustre: server umount lustre-MDT0001 complete [ 8930.479167] Lustre: server umount lustre-OST0000 complete [ 8943.598365] Lustre: server umount lustre-OST0001 complete [ 8958.437647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8971.934267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8980.895443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 8990.735045] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9013.417927] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 9024.674742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9024.707525] Lustre: lustre-MDT0000: reset Object Index mappings [ 9025.030054] LustreError: 260062:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9025.057316] LustreError: 260062:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 9025.249146] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9029.978622] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 9030.630641] LustreError: 260063:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9035.761833] LustreError: 260062:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9038.495345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9038.510482] Lustre: lustre-MDT0001: reset Object Index mappings [ 9038.930705] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9043.001142] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 9045.764988] Lustre: 261203:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9052.727569] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9059.535672] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 9063.465575] LustreError: 262090:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9068.528665] LustreError: 261558:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9069.013464] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9069.245794] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9069.253043] Lustre: Skipped 1 previous similar message [ 9070.273341] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9070.274256] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9076.348254] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 9083.576674] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9086.950830] Lustre: 263076:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9091.046111] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9091.050939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9091.072668] Lustre: Skipped 3 previous similar messages [ 9091.080808] Lustre: Skipped 3 previous similar messages [ 9096.167730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9096.174985] Lustre: Skipped 3 previous similar messages [ 9097.274242] Lustre: server umount lustre-MDT0000 complete [ 9100.509817] LustreError: 260044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788822822 with bad export cookie 15810612694116734166 [ 9100.511041] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9100.525444] LustreError: 260044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9100.877770] Lustre: server umount lustre-MDT0001 complete [ 9115.055875] Lustre: server umount lustre-OST0000 complete [ 9117.158249] Lustre: 250050:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788822823/real 1788822823] req@ffff9630c344c000 x1875716308334720/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788822839 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9117.202287] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9117.218891] Lustre: Skipped 1 previous similar message [ 9118.852290] Lustre: server umount lustre-OST0001 complete [ 9139.631723] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 19:14:19 (1788822859) [ 9156.124672] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_hostid [ 9165.242856] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 9206.624766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9207.055291] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9207.097850] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9207.204944] Lustre: lustre-MDT0000: new disk, initializing [ 9207.293136] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9207.317813] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9213.023760] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 9225.418956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9225.634162] Lustre: 267326:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9225.719205] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9225.723822] Lustre: Skipped 1 previous similar message [ 9225.915221] Lustre: lustre-MDT0001: new disk, initializing [ 9226.117758] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9226.132069] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9231.774152] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 9242.028088] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9247.442890] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9254.791340] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9255.082262] Lustre: lustre-OST0000: new disk, initializing [ 9255.089562] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9255.095413] Lustre: 269299:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9257.120095] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9257.142725] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9257.330984] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9262.092778] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 9272.789975] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9278.338891] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9278.572566] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9283.385276] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9283.656429] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9288.162961] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9288.170962] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9288.187796] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9288.195943] Lustre: Skipped 1 previous similar message [ 9292.162819] Lustre: server umount lustre-OST0000 complete [ 9297.898684] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9297.909111] LustreError: Skipped 1 previous similar message [ 9297.919203] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9297.938314] Lustre: Skipped 1 previous similar message [ 9297.945510] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9302.803821] Lustre: server umount lustre-MDT0000 complete [ 9303.523483] LustreError: 267338:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9303.548833] LustreError: 267338:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9306.786947] LustreError: 267320:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788823028 with bad export cookie 15810612694116736266 [ 9306.800300] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9306.816165] LustreError: 267320:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9307.450975] Lustre: server umount lustre-MDT0001 complete [ 9331.443776] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 9343.390687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9343.917788] LustreError: 272233:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9343.976458] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9343.979514] Lustre: Skipped 2 previous similar messages [ 9348.816310] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 9358.482170] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9364.952072] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 9369.053994] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9378.694889] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9388.540757] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 9399.213315] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9399.400572] Lustre: lustre-OST0001: new disk, initializing [ 9399.407655] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9399.413694] Lustre: 274541:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9400.699149] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9400.705254] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9400.866785] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9406.263530] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 9415.862730] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9419.075161] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9423.667164] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 9423.689661] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 9435.622243] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9435.638397] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9435.661073] Lustre: Skipped 1 previous similar message [ 9435.673684] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9435.679695] Lustre: Skipped 2 previous similar messages [ 9439.006097] Lustre: server umount lustre-MDT0000 complete [ 9441.764638] LustreError: 272234:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9441.793552] LustreError: 272234:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 9442.757629] LustreError: 272213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788823164 with bad export cookie 15810612694116737148 [ 9442.772801] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9442.774572] LustreError: 272213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9443.118896] Lustre: server umount lustre-MDT0001 complete [ 9459.076171] Lustre: server umount lustre-OST0000 complete [ 9462.242284] Lustre: 250051:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788823168/real 1788823168] req@ffff9631ec23e680 x1875716308444928/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788823184 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9462.705711] Lustre: server umount lustre-OST0001 complete [ 9481.346478] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 9496.631549] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9497.538918] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9497.550410] Lustre: Skipped 3 previous similar messages [ 9505.624091] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 9507.814422] LustreError: 277884:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9507.854764] LustreError: 277884:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9516.350857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9521.889267] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 9524.910613] Lustre: 279026:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9532.500582] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9540.787227] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 9542.470239] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 9550.472566] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9555.956802] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 9557.987486] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 9565.547856] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9569.726834] Lustre: 280904:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9575.375961] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 9575.513552] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 9589.539767] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 19:21:49 (1788823309) [ 9607.139382] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9607.140582] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9607.162666] Lustre: Skipped 7 previous similar messages [ 9607.179221] Lustre: Skipped 7 previous similar messages [ 9612.545694] Lustre: server umount lustre-MDT0000 complete [ 9616.823769] LustreError: 280906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788823338 with bad export cookie 15810612694116743777 [ 9616.839161] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9616.841030] LustreError: 280906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9631.200545] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9631.690608] Lustre: server umount lustre-MDT0001 complete [ 9643.972728] Lustre: server umount lustre-OST0000 complete [ 9648.893320] Lustre: server umount lustre-OST0001 complete [ 9655.458415] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_hostid [ 9669.426459] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 9719.541442] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 9733.942065] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9734.508189] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9734.543697] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9734.791414] Lustre: lustre-MDT0000: new disk, initializing [ 9735.000384] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9742.280634] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 9756.711464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9756.890198] Lustre: 285438:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9757.258534] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9757.261521] Lustre: Skipped 4 previous similar messages [ 9757.334848] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9762.844071] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 9768.306316] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9776.345492] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9776.685298] Lustre: 287072:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9778.425454] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9784.561045] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 9798.006293] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9798.253687] Lustre: 287944:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9799.643651] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9799.657742] Lustre: Skipped 1 previous similar message [ 9799.798933] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9805.997579] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 9817.307870] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9837.065883] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 19:25:57 (1788823557) [ 9840.623775] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9840.628069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9840.658767] Lustre: Skipped 5 previous similar messages [ 9840.677944] Lustre: Skipped 8 previous similar messages [ 9842.864859] Lustre: server umount lustre-MDT0000 complete [ 9845.745160] LustreError: 285446:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9845.779834] LustreError: 285446:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 19 previous similar messages [ 9847.267281] LustreError: 285430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788823569 with bad export cookie 15810612694116751491 [ 9847.274136] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9847.279506] LustreError: 285430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9847.719913] Lustre: server umount lustre-MDT0001 complete [ 9863.181686] Lustre: server umount lustre-OST0000 complete [ 9866.210458] Lustre: 250050:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788823572/real 1788823572] req@ffff9631fc961500 x1875716308602624/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788823588 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9868.259559] Lustre: 250051:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788823574/real 1788823574] req@ffff9631f1d56a00 x1875716308602880/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788823590 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9869.916638] Lustre: server umount lustre-OST0001 complete [ 9880.320047] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9885.783294] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 9894.696228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9899.874491] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 9907.904912] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9915.887712] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9924.731493] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9932.304721] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 9940.300917] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9959.639447] loop1: detected capacity change from 0 to 204800000 [ 9960.047743] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9960.083913] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9960.118590] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9967.708125] loop1: detected capacity change from 0 to 204800000 [ 9975.192160] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9975.316486] LustreError: 290177:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs31811: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [ 9975.325810] LustreError: 290177:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs31811 [ 9975.330159] LustreError: 290177:0:(mgs_handler.c:517:mgs_target_reg()) fs31811-MDT0001: Error checking target index: rc = -13 [ 9975.340899] LustreError: fs31811-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 9975.353162] LustreError: 293912:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -13 [ 9975.368686] LustreError: 293912:0:(tgt_mount.c:2128:server_put_super()) no obd fs31811-MDT0001 [ 9975.375113] LustreError: 293912:0:(tgt_mount.c:116:server_deregister_mount()) fs31811-MDT0001 not registered [ 9975.434343] Lustre: server umount fs31811-MDT0001 complete [ 9975.438753] LustreError: 293912:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 9986.047155] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9986.169738] Lustre: Setting parameter fs31811-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs31811-MDT0001 [ 9986.176797] Lustre: Skipped 1 previous similar message [ 9986.239543] Lustre: srv-fs31811-MDT0001: No data found on store. Initialize space. [ 9986.243850] Lustre: Skipped 4 previous similar messages [ 9986.297703] Lustre: fs31811-MDT0001: new disk, initializing [ 9986.300279] Lustre: Skipped 3 previous similar messages [ 9986.393271] Lustre: srv-fs31811-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 9986.399507] Lustre: 294379:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs31811-MDT0001-mdtlov: can't create new llog: rc = -115 [ 9986.402959] LustreError: 294379:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs31811-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 9992.420067] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [10002.000802] Lustre: server umount fs31811-MDT0001 complete [10007.023347] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10017.299986] loop1: detected capacity change from 0 to 204800000 [10022.390211] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10022.995869] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10023.573243] Lustre: 295948:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs31811-MDT0001-mdtlov: can't create new llog: rc = -115 [10023.583377] LustreError: 295948:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs31811-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10028.463283] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [10034.476295] Lustre: server umount fs31811-MDT0001 complete [10047.976516] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10047.977807] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10047.990449] Lustre: Skipped 2 previous similar messages [10047.999857] Lustre: Skipped 1 previous similar message [10050.782115] Lustre: server umount lustre-MDT0000 complete [10059.804298] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10059.972689] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10065.362223] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [10065.393690] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10074.579732] loop1: detected capacity change from 0 to 204800000 [10082.211933] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10082.856152] Lustre: 298395:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs31811-MDT0001-mdtlov: can't create new llog: rc = -115 [10082.868551] LustreError: 298395:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs31811-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10087.242170] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [10092.146722] Lustre: server umount fs31811-MDT0001 complete [10104.749341] LustreError: 292019:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10104.786575] LustreError: 292019:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 109 previous similar messages [10119.649331] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10125.746092] Lustre: server umount lustre-OST0000 complete [10136.766487] Lustre: server umount lustre-MDT0000 complete [10141.689924] LustreError: 290168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788823863 with bad export cookie 15810612694116754172 [10141.703212] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10141.707154] LustreError: 290168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10142.349417] Lustre: server umount lustre-MDT0001 complete [10158.976331] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [10163.422812] Key type lgssc unregistered [10163.901776] LNet: 300739:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10163.927914] LNetError: 300739:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10163.941718] LNet: Removed LNI 192.168.202.139@tcp [10165.237141] Key type .llcrypt unregistered [10165.238850] Key type ._llcrypt unregistered [10189.950782] Key type ._llcrypt registered [10189.954232] Key type .llcrypt registered [10190.134080] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [10192.042928] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10192.069339] alg: No test for adler32 (adler32-zlib) [10193.195658] Lustre: Lustre: Build Version: 2.17.58_40_gea5fd3f [10193.515868] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [10195.216238] Key type lgssc registered [10196.688868] Lustre: Echo OBD driver; http://www.lustre.org/ [10204.721341] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10204.745561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10206.345364] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10210.826849] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [10216.945137] LustreError: 302007:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10216.969480] LustreError: 302007:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10220.728761] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10221.264968] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10226.315644] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [10235.215856] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10242.349668] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10249.546413] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10250.035680] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10251.067593] LustreError: 303838:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10256.355788] LustreError: 303839:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10257.060927] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [10261.484294] LustreError: 303838:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10266.605106] LustreError: 303839:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10266.633623] LustreError: 303839:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10267.721481] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10275.747691] LustreError: 303840:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10275.780448] LustreError: 303840:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10287.208601] loop1: detected capacity change from 0 to 204800000 [10287.407113] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10287.419311] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10287.439679] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10292.199272] LustreError: 303839:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10292.216822] LustreError: 303839:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [10293.372706] loop1: detected capacity change from 0 to 204800000 [10299.105415] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10299.295986] Lustre: fs22804-OST0000: new disk, initializing [10299.301950] Lustre: srv-fs22804-OST0000: No data found on store. Initialize space. [10299.319283] Lustre: 305747:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10299.435234] Lustre: fs22804-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10306.835136] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [10315.417418] Lustre: server umount fs22804-OST0000 complete [10326.943446] LustreError: 303838:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10326.967488] LustreError: 303838:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 18 previous similar messages [10328.047652] loop1: detected capacity change from 0 to 204800000 [10333.161813] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10334.050290] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10334.217508] Lustre: fs22804-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10342.783717] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [10348.540632] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10352.488401] loop2: detected capacity change from 0 to 204800000 [10352.864541] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10352.871056] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10352.881358] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10358.612442] loop2: detected capacity change from 0 to 204800000 [10364.593754] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10364.682327] LustreError: fs22804-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [10364.703935] LustreError: fs22804-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10364.710540] LustreError: 308788:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -13 [10364.716434] LustreError: 308788:0:(tgt_mount.c:2128:server_put_super()) no obd fs22804-OST0001 [10364.720787] LustreError: 308788:0:(tgt_mount.c:116:server_deregister_mount()) fs22804-OST0001 not registered [10364.757632] Lustre: server umount fs22804-OST0001 complete [10364.760095] LustreError: 308788:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10376.072854] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10376.208992] Lustre: fs22804-OST0001: new disk, initializing [10376.212096] Lustre: srv-fs22804-OST0001: No data found on store. Initialize space. [10376.217771] Lustre: 309339:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10376.296024] Lustre: fs22804-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10384.064975] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [10393.371216] Lustre: server umount fs22804-OST0000 complete [10393.512778] LustreError: 303840:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10393.537283] LustreError: 303840:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 38 previous similar messages [10403.906594] Lustre: server umount fs22804-OST0001 complete [10418.310970] loop1: detected capacity change from 0 to 204800000 [10425.544625] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10426.032215] Lustre: fs22804-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10433.124724] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [10440.718862] loop2: detected capacity change from 0 to 204800000 [10447.285375] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10455.348728] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [10466.069299] Lustre: server umount fs22804-OST0000 complete [10477.528702] Lustre: server umount fs22804-OST0001 complete [10486.770440] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10496.993363] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10497.009118] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10497.031199] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10500.885887] Lustre: server umount lustre-OST0000 complete [10507.235520] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10507.239606] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10507.245954] Lustre: Skipped 2 previous similar messages [10507.260779] Lustre: Skipped 2 previous similar messages [10511.634521] Lustre: server umount lustre-MDT0000 complete [10515.981591] LustreError: 301989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788824237 with bad export cookie 17545180500326840796 [10515.990146] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10515.996432] LustreError: 301989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10516.568724] Lustre: server umount lustre-MDT0001 complete [10530.604306] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [10535.226707] Key type lgssc unregistered [10535.690668] LNet: 315266:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10535.719782] LNetError: 315266:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10535.767390] LNet: Removed LNI 192.168.202.139@tcp [10536.965156] Key type .llcrypt unregistered [10536.967314] Key type ._llcrypt unregistered [10575.344614] Key type ._llcrypt registered [10575.345825] Key type .llcrypt registered [10575.478823] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_hostid [10591.807250] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [10593.177256] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10593.242048] alg: No test for adler32 (adler32-zlib) [10594.382729] Lustre: Lustre: Build Version: 2.17.58_40_gea5fd3f [10594.703396] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [10596.408518] Key type lgssc registered [10597.617854] Lustre: Echo OBD driver; http://www.lustre.org/ [10633.122113] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10633.147086] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10634.525667] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10634.571483] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10634.689734] Lustre: lustre-MDT0000: new disk, initializing [10634.829231] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10634.845370] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10639.132743] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [10648.945803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10649.108841] Lustre: 319125:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10649.164470] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10649.171520] Lustre: Skipped 1 previous similar message [10649.249518] Lustre: lustre-MDT0001: new disk, initializing [10649.394760] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10649.448557] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10649.456840] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10654.915795] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [10664.081828] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10669.472031] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10676.649324] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10676.956449] Lustre: lustre-OST0000: new disk, initializing [10676.961144] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10676.971429] Lustre: 321093:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10677.068033] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10682.934895] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10682.951263] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10683.100348] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10684.092549] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [10692.980568] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10695.006960] Lustre: Mounted lustre-client - version 2.17.58_40_gea5fd3f [10695.987273] Lustre: Unmounted lustre-client [10705.378730] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10705.386439] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10705.389424] Lustre: Skipped 1 previous similar message [10710.500027] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10710.509057] Lustre: Skipped 2 previous similar messages [10715.619412] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10715.635217] Lustre: Skipped 1 previous similar message [10719.200268] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10719.455468] Lustre: server umount lustre-OST0000 complete [10725.859170] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10725.879464] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10725.880038] Lustre: Skipped 1 previous similar message [10725.895228] Lustre: Skipped 1 previous similar message [10729.645138] Lustre: server umount lustre-MDT0000 complete [10732.652759] LustreError: 319119:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788824454 with bad export cookie 8483823739250518810 [10732.660168] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10733.012952] Lustre: server umount lustre-MDT0001 complete [10742.193589] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [10744.553883] Key type lgssc unregistered [10750.368387] LNet: 1 local NIs in recovery (showing 1): 192.168.202.139@tcp [10751.009897] LNet: 323254:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10751.034717] LNetError: 323254:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10826.859199] LNet: Removed LNI 192.168.202.139@tcp [10827.766058] Key type .llcrypt unregistered [10827.767326] Key type ._llcrypt unregistered [10841.580938] Key type ._llcrypt registered [10841.582717] Key type .llcrypt registered [10841.659308] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_hostid [10853.252434] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [10854.039553] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10854.063458] alg: No test for adler32 (adler32-zlib) [10855.022685] Lustre: Lustre: Build Version: 2.17.58_40_gea5fd3f [10855.207491] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [10856.840185] Key type lgssc registered [10857.749735] Lustre: Echo OBD driver; http://www.lustre.org/ [10879.008590] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 19:43:20 (1788824600) [10883.319040] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10883.341454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10884.490411] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10884.509668] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10884.559436] Lustre: lustre-MDT0000: new disk, initializing [10884.594358] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10884.605854] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10887.971996] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [10896.637859] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10896.765778] Lustre: 326452:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10896.821702] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10896.825665] Lustre: Skipped 1 previous similar message [10896.914967] Lustre: lustre-MDT0001: new disk, initializing [10897.006562] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10897.042311] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10897.053407] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10901.135977] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [10908.905641] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10914.771848] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10920.779407] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10920.983352] Lustre: lustre-OST0000: new disk, initializing [10920.986898] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10920.993481] Lustre: 328422:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10921.072866] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10926.199632] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [10928.173065] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10928.190110] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10928.287021] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10934.193352] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10948.577694] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10948.583819] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10948.598525] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10949.096078] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10949.104658] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10949.118923] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10949.122189] Lustre: Skipped 1 previous similar message [10952.227983] Lustre: server umount lustre-OST0000 complete [10958.306413] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10958.317980] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10958.335089] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10961.243769] Lustre: server umount lustre-MDT0000 complete [10964.562772] LustreError: 326444:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788824686 with bad export cookie 8859171939761532025 [10964.564251] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10964.568336] LustreError: 326444:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10964.853298] Lustre: server umount lustre-MDT0001 complete [10973.855613] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [10975.869755] Key type lgssc unregistered [10976.104836] LNet: 330328:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10976.112555] LNetError: 330328:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10976.141882] LNet: Removed LNI 192.168.202.139@tcp [10976.859345] Key type .llcrypt unregistered [10976.861276] Key type ._llcrypt unregistered [10992.660375] Key type ._llcrypt registered [10992.662552] Key type .llcrypt registered [10992.746645] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [10993.843292] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10993.941381] alg: No test for adler32 (adler32-zlib) [10995.014447] Lustre: Lustre: Build Version: 2.17.58_40_gea5fd3f [10995.170940] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [10996.808226] Key type lgssc registered [10997.771631] Lustre: Echo OBD driver; http://www.lustre.org/ [11003.643520] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11003.663593] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11004.936911] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11007.871619] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [11013.744935] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11014.033287] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11017.222752] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [11023.049787] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11028.133779] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11034.476890] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11034.762808] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11040.615435] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [11045.884873] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11058.402370] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11059.503939] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11062.742947] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.39@tcp (stopping) [11065.313502] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11065.324815] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11065.345851] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11065.829995] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11065.847900] Lustre: Skipped 1 previous similar message [11067.820941] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.39@tcp (stopping) [11067.824887] Lustre: Skipped 2 previous similar messages [11068.630737] Lustre: server umount lustre-MDT0000 complete [11070.951457] LustreError: 332306:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11070.962696] LustreError: 332306:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [11071.447958] LustreError: 331559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788824793 with bad export cookie 13981646092337402828 [11071.453995] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11071.458685] LustreError: 331559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11072.925104] LustreError: 331575:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11072.946794] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.39@tcp (stopping) [11076.071479] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11076.072362] LustreError: 331574:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11077.776947] Lustre: server umount lustre-MDT0001 complete [11086.550858] Lustre: server umount lustre-OST0000 complete [11092.517629] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11095.591548] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [11097.419760] Lustre: 335756:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.139@tcp [11098.808541] Lustre: server umount MGS complete [11105.188383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11105.509589] LustreError: 336327:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11105.550564] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11108.794927] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [11110.884420] LustreError: 336328:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11115.826535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11116.006387] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [11116.012943] Lustre: Skipped 1 previous similar message [11116.078430] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11119.101676] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [11126.122221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11126.244775] LustreError: 337760:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11126.258493] LustreError: 337760:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11126.380983] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11130.940789] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [11135.889346] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11137.177558] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [11141.629751] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11142.844933] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11146.213807] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11146.218906] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11146.222533] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11146.228400] Lustre: Skipped 1 previous similar message [11146.241505] Lustre: Skipped 1 previous similar message [11152.005131] Lustre: server umount lustre-OST0000 complete [11156.449949] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11156.457875] Lustre: Skipped 1 previous similar message [11166.693901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11166.699791] Lustre: Skipped 7 previous similar messages [11169.249160] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11169.425202] Lustre: server umount lustre-MDT0000 complete [11171.810145] LustreError: 337066:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11171.825424] LustreError: 337066:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11172.420929] LustreError: 336309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788824894 with bad export cookie 13981646092337403668 [11172.422737] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11172.432561] LustreError: 336309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [11172.747773] Lustre: server umount lustre-MDT0001 complete [11180.702472] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [11182.698167] Key type lgssc unregistered [11182.987547] LNet: 339760:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11182.994952] LNetError: 339760:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11183.024200] LNet: Removed LNI 192.168.202.139@tcp [11183.643709] Key type .llcrypt unregistered [11183.646885] Key type ._llcrypt unregistered [11196.250290] Key type ._llcrypt registered [11196.251523] Key type .llcrypt registered [11196.319303] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_hostid [11207.609924] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [11208.219263] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11208.228953] alg: No test for adler32 (adler32-zlib) [11209.293257] Lustre: Lustre: Build Version: 2.17.58_40_gea5fd3f [11209.525216] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [11211.128204] Key type lgssc registered [11212.061467] Lustre: Echo OBD driver; http://www.lustre.org/ [11237.142465] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11237.156237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11238.393834] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11238.419600] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11238.481822] Lustre: lustre-MDT0000: new disk, initializing [11238.536339] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11238.551878] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11241.800912] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [11249.253217] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11249.347396] Lustre: 342894:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11249.386901] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11249.390439] Lustre: Skipped 1 previous similar message [11249.438456] Lustre: lustre-MDT0001: new disk, initializing [11249.513336] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11249.546353] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11249.557851] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11252.766140] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [11260.092051] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11264.170903] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11269.195672] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11269.423857] Lustre: lustre-OST0000: new disk, initializing [11269.428143] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11269.435272] Lustre: 344862:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11269.511160] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11273.734847] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [11277.360769] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11277.367499] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11277.471873] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11280.910264] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11283.986254] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11284.108199] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11288.153482] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11288.324466] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11293.154294] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11293.166148] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11293.167768] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11293.173722] LustreError: Skipped 1 previous similar message [11293.181138] Lustre: Skipped 1 previous similar message [11295.961264] Lustre: server umount lustre-OST0000 complete [11300.835987] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11300.843932] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11300.857225] Lustre: Skipped 1 previous similar message [11300.867421] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11303.394545] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11303.394721] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11305.460562] Lustre: server umount lustre-MDT0000 complete [11308.588544] LustreError: 344861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788825030 with bad export cookie 6004433855349861812 [11308.595310] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11308.596716] LustreError: 344861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11308.918555] Lustre: server umount lustre-MDT0001 complete [11316.291064] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 19:50:37 (1788825037) [11330.547405] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_hostid [11337.847612] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [11367.011628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11367.222918] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11367.245492] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11367.301840] Lustre: lustre-MDT0000: new disk, initializing [11367.361324] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11367.378726] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11370.727724] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [11379.408780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11379.496808] Lustre: 349618:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11379.543190] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11379.546713] Lustre: Skipped 1 previous similar message [11379.605337] Lustre: lustre-MDT0001: new disk, initializing [11379.668518] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11379.691101] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11379.701745] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11382.672654] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [11389.515899] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11393.043800] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11397.696474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11397.887435] Lustre: lustre-OST0000: new disk, initializing [11397.891375] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11397.895424] Lustre: 351582:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11397.962386] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11402.492807] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [11403.806081] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11403.816031] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11403.891395] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11409.360477] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11429.361107] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11429.365569] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11429.375524] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11429.388502] Lustre: Skipped 1 previous similar message [11429.397864] Lustre: Skipped 2 previous similar messages [11434.337412] Lustre: server umount lustre-OST0000 complete [11439.586021] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11439.588732] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11439.601216] Lustre: Skipped 1 previous similar message [11449.827790] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11449.832580] Lustre: Skipped 4 previous similar messages [11451.360223] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11451.536640] Lustre: server umount lustre-MDT0000 complete [11454.493455] LustreError: 349610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788825176 with bad export cookie 6004433855349862694 [11454.503208] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11454.503680] LustreError: 349610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11454.690213] Lustre: server umount lustre-MDT0001 complete [11461.701254] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [11463.509197] Key type lgssc unregistered [11463.717785] LNet: 353731:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11463.725585] LNetError: 353731:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11463.741817] LNet: Removed LNI 192.168.202.139@tcp [11464.376952] Key type .llcrypt unregistered [11464.378811] Key type ._llcrypt unregistered [11531.780098] Key type ._llcrypt registered [11531.781821] Key type .llcrypt registered [11531.903331] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [11532.154371] Key type .llcrypt unregistered [11532.158963] Key type ._llcrypt unregistered [11586.406286] Key type ._llcrypt registered [11586.408065] Key type .llcrypt registered [11586.493834] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [11586.687742] Key type .llcrypt unregistered [11586.691206] Key type ._llcrypt unregistered [11607.748386] Key type ._llcrypt registered [11607.750353] Key type .llcrypt registered [11607.802627] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [11607.994512] Key type .llcrypt unregistered [11607.996415] Key type ._llcrypt unregistered [11628.636754] Key type ._llcrypt registered [11628.640753] Key type .llcrypt registered [11628.730831] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [11628.914822] Key type .llcrypt unregistered [11628.917278] Key type ._llcrypt unregistered [11679.033821] Key type ._llcrypt registered [11679.038612] Key type .llcrypt registered [11679.131933] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [11679.310745] Key type .llcrypt unregistered [11679.314136] Key type ._llcrypt unregistered [11699.274734] Key type ._llcrypt registered [11699.276410] Key type .llcrypt registered [11699.356770] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [11699.531259] Key type .llcrypt unregistered [11699.533766] Key type ._llcrypt unregistered [11719.278263] Key type ._llcrypt registered [11719.280961] Key type .llcrypt registered [11719.382982] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [11719.593476] Key type .llcrypt unregistered [11719.595115] Key type ._llcrypt unregistered [11739.393768] Key type ._llcrypt registered [11739.396023] Key type .llcrypt registered [11739.466434] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [11739.644645] Key type .llcrypt unregistered [11739.647255] Key type ._llcrypt unregistered [11762.966399] Key type ._llcrypt registered [11762.971966] Key type .llcrypt registered [11763.039034] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [11763.250413] Key type .llcrypt unregistered [11763.252964] Key type ._llcrypt unregistered [11782.989939] Key type ._llcrypt registered [11782.991215] Key type .llcrypt registered [11783.057971] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [11783.206386] Key type .llcrypt unregistered [11783.207659] Key type ._llcrypt unregistered [11820.335450] Key type ._llcrypt registered [11820.337147] Key type .llcrypt registered [11820.391907] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [11820.555203] Key type .llcrypt unregistered [11820.557833] Key type ._llcrypt unregistered [11839.332958] Key type ._llcrypt registered [11839.334668] Key type .llcrypt registered [11839.418584] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [11839.651289] Key type .llcrypt unregistered [11839.652790] Key type ._llcrypt unregistered [11859.552272] Key type ._llcrypt registered [11859.553862] Key type .llcrypt registered [11859.614527] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [11859.785301] Key type .llcrypt unregistered [11859.789926] Key type ._llcrypt unregistered [11887.857196] Key type ._llcrypt registered [11887.859230] Key type .llcrypt registered [11887.923928] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_hostid [11896.509877] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [11897.234551] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11897.277824] alg: No test for adler32 (adler32-zlib) [11898.221530] Lustre: Lustre: Build Version: 2.17.58_40_gea5fd3f [11898.379353] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [11900.010666] Key type lgssc registered [11900.691916] Lustre: Echo OBD driver; http://www.lustre.org/ [11921.059735] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11921.077513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11922.199482] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [11922.214552] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [11922.254377] Lustre: tst250fs-MDT0000: new disk, initializing [11922.287715] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11922.298321] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11924.562526] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [11931.069623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11931.161914] Lustre: 373621:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11931.207593] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [11931.210057] Lustre: tst250fs-MDT0001: Not available for connect from 0@lo (not set up) [11931.212849] Lustre: Skipped 1 previous similar message [11931.306854] Lustre: tst250fs-MDT0001: new disk, initializing [11931.363175] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11931.391994] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11931.397642] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11934.150585] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [11937.476322] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11941.988399] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11942.229596] Lustre: tst250fs-OST0000: new disk, initializing [11942.233396] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [11942.237988] Lustre: 375266:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11942.300589] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11946.642352] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [11947.063059] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11947.069749] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11947.137174] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11954.230596] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11954.388554] Lustre: tst250fs-OST0001: new disk, initializing [11954.392453] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [11954.398641] Lustre: 376137:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11954.491677] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11956.489111] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11956.493824] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11956.569716] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11959.029789] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [11962.984508] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [11977.697133] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11977.703298] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11977.715562] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [11982.321335] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11982.323528] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [11982.327695] Lustre: Skipped 1 previous similar message [11982.330301] Lustre: Skipped 1 previous similar message [11983.680046] Lustre: server umount tst250fs-MDT0000 complete [11986.060965] LustreError: 373613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788825708 with bad export cookie 8040238474905242042 [11986.063154] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11986.075775] LustreError: 373613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11986.274499] Lustre: server umount tst250fs-MDT0001 complete [11998.891803] Lustre: server umount tst250fs-OST0000 complete [12011.826275] Lustre: server umount tst250fs-OST0001 complete [12024.529526] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_hostid [12029.996741] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [12051.268461] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 20:02:52 (1788825772) [12052.366716] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [12064.258786] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_hostid [12070.608391] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [12091.006390] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 11817 sec ========= 20:03:32 (1788825812) [12092.211197] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 20:03:33 (1788825813) === [12094.844908] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 20:03:35 (1788825815) === [12118.810902] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [12120.344294] Key type lgssc unregistered [12120.498906] LNet: 385262:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12120.502594] LNetError: 385262:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12120.512357] LNet: Removed LNI 192.168.202.139@tcp [12120.966113] Key type .llcrypt unregistered [12120.967170] Key type ._llcrypt unregistered