[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 444507267 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003124] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005013] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.010136] LSM: Security Framework initializing [ 0.011053] Yama: becoming mindful. [ 0.012038] SELinux: Initializing. [ 0.014050] *** VALIDATE selinux *** [ 0.022017] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026756] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027143] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029044] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031016] *** VALIDATE tmpfs *** [ 0.032427] *** VALIDATE proc *** [ 0.034049] *** VALIDATE cgroup *** [ 0.035008] *** VALIDATE cgroup2 *** [ 0.036266] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037143] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039029] Spectre V2 : User space: Vulnerable [ 0.040004] Speculative Store Bypass: Vulnerable [ 0.042648] debug: unmapping init [mem 0xffffffffa1259000-0xffffffffa1260fff] [ 0.044203] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045710] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046027] ... version: 2 [ 0.047009] ... bit width: 48 [ 0.048011] ... generic registers: 4 [ 0.048999] ... value mask: 0000ffffffffffff [ 0.050015] ... max period: 00007fffffffffff [ 0.051013] ... fixed-purpose events: 3 [ 0.051912] ... event mask: 000000070000000f [ 0.052293] rcu: Hierarchical SRCU implementation. [ 0.054384] smp: Bringing up secondary CPUs ... [ 0.055490] x86: Booting SMP configuration: [ 0.056016] .... node #0, CPUs: #1 #2 #3 [ 0.063009] smp: Brought up 1 node, 4 CPUs [ 0.065023] smpboot: Max logical packages: 1 [ 0.066007] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.105078] node 0 deferred pages initialised in 37ms [ 0.107395] devtmpfs: initialized [ 0.108264] x86/mm: Memory block size: 128MB [ 0.111067] gcov: version magic: 0x41383552 [ 0.113126] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.114081] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.115353] pinctrl core: initialized pinctrl subsystem [ 0.116193] [ 0.116572] ************************************************************* [ 0.117016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.118014] ** ** [ 0.119018] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.120011] ** ** [ 0.121012] ** This means that this kernel is built to expose internal ** [ 0.122013] ** IOMMU data structures, which may compromise security on ** [ 0.123012] ** your system. ** [ 0.124017] ** ** [ 0.125014] ** If you see this message and you are not debugging the ** [ 0.126015] ** kernel, report this immediately to your vendor! ** [ 0.127018] ** ** [ 0.128014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.129020] ************************************************************* [ 0.130712] NET: Registered protocol family 16 [ 0.131311] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.132052] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.133042] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.134608] cpuidle: using governor menu [ 0.135697] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.136406] PCI: Using configuration type 1 for base access [ 0.137106] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.144180] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.145074] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.146198] cryptd: max_cpu_qlen set to 1000 [ 0.148094] ACPI: Added _OSI(Module Device) [ 0.149011] ACPI: Added _OSI(Processor Device) [ 0.149913] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.150013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.152943] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.154533] ACPI: Interpreter enabled [ 0.155058] ACPI: PM: (supports S0 S3 S4 S5) [ 0.155796] ACPI: Using IOAPIC for interrupt routing [ 0.156071] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.157276] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.165103] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.166033] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.167017] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.168078] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.169918] acpiphp: Slot [2] registered [ 0.170072] acpiphp: Slot [5] registered [ 0.171109] acpiphp: Slot [6] registered [ 0.171958] acpiphp: Slot [7] registered [ 0.172058] acpiphp: Slot [8] registered [ 0.173149] acpiphp: Slot [9] registered [ 0.174097] acpiphp: Slot [10] registered [ 0.175120] acpiphp: Slot [3] registered [ 0.176126] acpiphp: Slot [4] registered [ 0.177104] acpiphp: Slot [11] registered [ 0.178116] acpiphp: Slot [12] registered [ 0.179159] acpiphp: Slot [13] registered [ 0.180150] acpiphp: Slot [14] registered [ 0.181112] acpiphp: Slot [15] registered [ 0.182124] acpiphp: Slot [16] registered [ 0.183100] acpiphp: Slot [17] registered [ 0.183976] acpiphp: Slot [18] registered [ 0.184106] acpiphp: Slot [19] registered [ 0.185042] acpiphp: Slot [20] registered [ 0.186086] acpiphp: Slot [21] registered [ 0.186993] acpiphp: Slot [22] registered [ 0.187059] acpiphp: Slot [23] registered [ 0.187921] acpiphp: Slot [24] registered [ 0.188057] acpiphp: Slot [25] registered [ 0.188813] acpiphp: Slot [26] registered [ 0.189092] acpiphp: Slot [27] registered [ 0.190105] acpiphp: Slot [28] registered [ 0.191081] acpiphp: Slot [29] registered [ 0.192042] acpiphp: Slot [30] registered [ 0.192911] acpiphp: Slot [31] registered [ 0.193045] PCI host bridge to bus 0000:00 [ 0.194018] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.195032] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.196026] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.197040] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.198032] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.199027] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.200186] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.202077] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.203885] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.208020] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.211031] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.212015] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.213010] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.214010] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.215409] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.216592] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.217050] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.218727] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.221014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.230018] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.232803] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.236419] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.239016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.242014] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.249016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.254742] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.257014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.260026] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.267016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.272654] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.275015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.278013] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.286020] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.292577] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.295013] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.299013] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.306017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.313642] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.316044] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.319049] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.326016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.332424] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.344018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.362022] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.381019] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.392659] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.395334] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.398417] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.404306] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.407191] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.412125] iommu: Default domain type: Passthrough [ 0.415769] SCSI subsystem initialized [ 0.417150] ACPI: bus type USB registered [ 0.419129] usbcore: registered new interface driver usbfs [ 0.422089] usbcore: registered new interface driver hub [ 0.424139] usbcore: registered new device driver usb [ 0.427315] pps_core: LinuxPPS API ver. 1 registered [ 0.429244] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.433057] PTP clock support registered [ 0.435169] EDAC MC: Ver: 3.0.0 [ 0.436000] PCI: Using ACPI for IRQ routing [ 0.437963] NetLabel: Initializing [ 0.439013] NetLabel: domain hash size = 128 [ 0.441010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.442122] NetLabel: unlabeled traffic allowed by default [ 0.445094] vgaarb: loaded [ 0.446353] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.448062] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.455872] clocksource: Switched to clocksource kvm-clock [ 0.547089] VFS: Disk quotas dquot_6.6.0 [ 0.549210] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.553226] *** VALIDATE ramfs *** [ 0.555273] *** VALIDATE hugetlbfs *** [ 0.557985] pnp: PnP ACPI init [ 0.561175] pnp: PnP ACPI: found 6 devices [ 0.579675] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.583620] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.586261] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.589044] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.592898] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.595316] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.598614] NET: Registered protocol family 2 [ 0.601456] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.607710] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.611687] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.617244] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.623231] TCP: Hash tables configured (established 65536 bind 65536) [ 0.627212] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.631156] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.635158] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.639039] NET: Registered protocol family 1 [ 0.641900] RPC: Registered named UNIX socket transport module. [ 0.645238] RPC: Registered udp transport module. [ 0.647870] RPC: Registered tcp transport module. [ 0.650424] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.654182] NET: Registered protocol family 44 [ 0.655754] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.658243] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.660076] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.662137] PCI: CLS 0 bytes, default 64 [ 0.663608] Unpacking initramfs... [ 2.090281] debug: unmapping init [mem 0xffff8f053cc54000-0xffff8f053ffbffff] [ 2.093562] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.094839] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.096733] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.602086] Initialise system trusted keyrings [ 2.603459] Key type blacklist registered [ 2.605250] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.613700] zbud: loaded [ 2.616852] *** VALIDATE nfs *** [ 2.617881] *** VALIDATE nfs4 *** [ 2.619183] pstore: using deflate compression [ 2.622404] Platform Keyring initialized [ 2.716532] NET: Registered protocol family 38 [ 2.718165] Key type asymmetric registered [ 2.719666] Asymmetric key parser 'x509' registered [ 2.721937] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.725322] io scheduler mq-deadline registered [ 2.727127] io scheduler kyber registered [ 2.728806] io scheduler bfq registered [ 2.731044] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.734421] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.736740] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.739032] ACPI: Power Button [PWRF] [ 2.744135] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.751261] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.769817] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.777514] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.792739] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.820927] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.850219] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.854677] Non-volatile memory driver v1.3 [ 2.856120] Linux agpgart interface v0.103 [ 2.886210] virtio_blk virtio1: [vda] 149960 512-byte logical blocks (76.8 MB/73.2 MiB) [ 2.888507] vda: detected capacity change from 0 to 76779520 [ 2.902453] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.905086] vdb: detected capacity change from 0 to 1073741824 [ 2.919764] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.922474] vdc: detected capacity change from 0 to 2621440000 [ 2.937643] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.940233] vdd: detected capacity change from 0 to 2621440000 [ 2.957375] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.960435] vde: detected capacity change from 0 to 4294967296 [ 2.974136] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.978239] vdf: detected capacity change from 0 to 4294967296 [ 2.990982] libphy: Fixed MDIO Bus: probed [ 3.001549] usbcore: registered new interface driver usbserial_generic [ 3.004410] usbserial: USB Serial support registered for generic [ 3.007253] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.012270] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.014565] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.017842] mousedev: PS/2 mouse device common for all mice [ 3.021158] rtc_cmos 00:05: RTC can wake from S4 [ 3.023119] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.026686] rtc_cmos 00:05: registered as rtc0 [ 3.032206] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.032443] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.037803] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.038925] intel_pstate: CPU model not supported [ 3.044149] hid: raw HID events driver (C) Jiri Kosina [ 3.045433] usbcore: registered new interface driver usbhid [ 3.046767] usbhid: USB HID core driver [ 3.047718] drop_monitor: Initializing network drop monitor service [ 3.049887] Initializing XFRM netlink socket [ 3.051543] NET: Registered protocol family 10 [ 3.054039] Segment Routing with IPv6 [ 3.055819] NET: Registered protocol family 17 [ 3.057942] mpls_gso: MPLS GSO support [ 3.064035] RAS: Correctable Errors collector initialized. [ 3.066437] AVX version of gcm_enc/dec engaged. [ 3.068240] AES CTR mode by8 optimization enabled [ 3.140581] sched_clock: Marking stable (3140559763, 0)->(4252092588, -1111532825) [ 3.143625] registered taskstats version 1 [ 3.145213] Loading compiled-in X.509 certificates [ 3.146598] zswap: loaded using pool lzo/zbud [ 3.163073] Key type big_key registered [ 3.171756] Key type encrypted registered [ 3.173061] ima: No TPM chip found, activating TPM-bypass! [ 3.174805] ima: Allocated hash algorithm: sha1 [ 3.176072] ima: No architecture policies found [ 3.177395] evm: Initialising EVM extended attributes: [ 3.178902] evm: security.selinux [ 3.179982] evm: security.ima [ 3.180803] evm: security.capability [ 3.181783] evm: HMAC attrs: 0x1 [ 3.183764] rtc_cmos 00:05: setting system clock to 2026-09-07 17:21:34 UTC (1788801694) [ 3.189190] debug: unmapping init [mem 0xffffffffa2203000-0xffffffffa23fffff] [ 3.191890] debug: unmapping init [mem 0xffffffffa0f82000-0xffffffffa1258fff] [ 3.200403] Write protecting the kernel read-only data: 28672k [ 3.203258] debug: unmapping init [mem 0xffffffff9f603000-0xffffffff9f7fffff] [ 3.205438] debug: unmapping init [mem 0xffffffff9ff14000-0xffffffff9fffffff] [ 3.235454] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.242351] systemd[1]: Detected virtualization kvm. [ 3.243717] systemd[1]: Detected architecture x86-64. [ 3.244892] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.267312] systemd[1]: No hostname configured. [ 3.268671] systemd[1]: Set hostname to . [ 3.270576] random: systemd: uninitialized urandom read (16 bytes read) [ 3.273049] systemd[1]: Initializing machine ID from random generator. [ 3.315681] random: ln: uninitialized urandom read (6 bytes read) [ 3.399283] random: systemd: uninitialized urandom read (16 bytes read) [ 3.401239] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.404520] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.407753] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Timers. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Slices. Starting Setup Virtual Console... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.959612] device-mapper: uevent: version 1.0.3 [ 3.961996] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. [ 4.239238] random: fast init done Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.638912] virtio_net virtio0 ens2: renamed from eth0 [ 4.709414] scsi host0: ata_piix [ 4.737793] scsi host1: ata_piix [ 4.739761] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.742065] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.107386] dracut-initqueue[588]: RTNETLINK answers: File exists [ 9.700971] random: crng init done [ 9.702340] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.138528] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.524310] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.802664] SELinux: Disabled at runtime. [ 11.866754] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.877390] systemd[1]: Detected virtualization kvm. [ 11.879770] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.418081] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.422320] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.427283] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.432325] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.436922] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.450175] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.458872] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on RPCbind Server Activation Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Kernel Debug File System... Mounting POSIX Message Queue File System... [ OK ] Created slice system-getty.slice. Mounting Huge Pages File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Starting Apply Kernel Variables... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on initctl Compatibility Named Pipe. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. Starting Remount Root a[ 12.632281] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS nd Kernel File Systems... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 12.936498] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.301749] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.315411] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.522965] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.533304] EDAC sbridge: Ver: 1.1.2 [ 15.188916] Key type dns_resolver registered [ 15.511227] NFS: Registering the id_resolver key type [ 15.513285] Key type id_resolver registered [ 15.514952] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started Login Service. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg104-server login: [ 65.462277] libcfs: loading out-of-tree module taints kernel. [ 65.493434] Key type ._llcrypt registered [ 65.495446] Key type .llcrypt registered [ 65.576439] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [ 81.140385] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 82.067117] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 82.083576] alg: No test for adler32 (adler32-zlib) [ 83.619819] Lustre: Lustre: Build Version: 2.17.58_39_gb44c962 [ 84.738564] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 86.503257] Key type lgssc registered [ 88.137943] Lustre: Echo OBD driver; http://www.lustre.org/ [ 104.769553] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 126.049418] hrtimer: interrupt took 5409302 ns [ 145.823152] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 159.109773] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 159.142554] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 160.438385] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 160.473619] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 160.623343] Lustre: lustre-MDT0000: new disk, initializing [ 160.757552] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 160.786759] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 165.397629] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 177.891068] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 177.989373] Lustre: 6509:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 178.031888] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 178.036986] Lustre: Skipped 1 previous similar message [ 178.157139] Lustre: lustre-MDT0001: new disk, initializing [ 178.244767] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 178.261042] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 178.271913] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 183.101900] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 187.987730] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 197.190164] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 197.373969] Lustre: lustre-OST0000: new disk, initializing [ 197.377361] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 197.382323] Lustre: 8445:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 197.439834] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 203.347247] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 206.353198] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 206.379546] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 206.444933] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 216.443361] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 216.589172] Lustre: lustre-OST0001: new disk, initializing [ 216.593668] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 216.600371] Lustre: 9517:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 216.670872] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 222.133489] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 223.793756] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 223.811685] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 223.908420] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 234.156271] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 241.765403] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 247.577832] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing check_logdir /tmp/testlogs/ [ 253.363211] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing yml_node [ 259.484253] Lustre: DEBUG MARKER: Client: 2.17.58.39 [ 263.909244] Lustre: DEBUG MARKER: MDS: 2.17.58.39 [ 267.396870] Lustre: DEBUG MARKER: OSS: 2.17.58.39 [ 269.152603] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Sep 7 13:26:00 EDT 2026 [ 286.958601] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 289.108915] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 310.751946] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 310.754305] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 310.774900] Lustre: Skipped 2 previous similar messages [ 310.795080] Lustre: Skipped 3 previous similar messages [ 312.715405] Lustre: server umount lustre-MDT0000 complete [ 315.874960] LustreError: 6502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788802007 with bad export cookie 16019378834491439943 [ 315.877401] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 315.878783] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 315.882822] LustreError: 6502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 315.921860] Lustre: Skipped 1 previous similar message [ 315.934639] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 320.998166] LustreError: 12259:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 321.005545] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 321.013112] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 321.037985] LustreError: 12259:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 326.112108] LustreError: 9815:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 326.118853] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 326.123143] LustreError: 9815:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 326.147274] Lustre: Skipped 2 previous similar messages [ 327.137700] LustreError: 12259:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 327.153553] LustreError: 12259:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 330.207553] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 330.520287] Lustre: server umount lustre-MDT0001 complete [ 334.472985] Lustre: server umount lustre-OST0000 complete [ 338.151294] Lustre: server umount lustre-OST0001 complete [ 344.472440] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [ 354.930782] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 393.714723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 393.981065] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 394.038777] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 394.182783] Lustre: lustre-MDT0000: new disk, initializing [ 394.236701] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 394.252550] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 399.164356] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 411.464996] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 411.645409] Lustre: 15078:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 411.719046] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 411.725454] Lustre: Skipped 1 previous similar message [ 411.819232] Lustre: lustre-MDT0001: new disk, initializing [ 411.966599] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 412.025036] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 412.046263] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 417.338992] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 428.931343] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 435.424764] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 441.016032] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 441.206104] Lustre: lustre-OST0000: new disk, initializing [ 441.209696] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 441.214694] Lustre: 17044:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 441.357461] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 442.331028] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 442.336603] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 442.408751] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 446.568393] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 454.905761] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 459.789603] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 460.133917] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 465.086673] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 465.303616] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 467.939531] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 467.940550] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 467.968867] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 467.982156] Lustre: Skipped 3 previous similar messages [ 474.051988] Lustre: server umount lustre-OST0000 complete [ 478.688988] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 478.696059] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 478.710883] Lustre: Skipped 1 previous similar message [ 478.720334] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 478.726206] Lustre: Skipped 2 previous similar messages [ 484.698773] Lustre: server umount lustre-MDT0000 complete [ 488.422251] LustreError: 15088:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 488.452390] LustreError: 15088:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 488.806213] LustreError: 15070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788802180 with bad export cookie 16019378834491441469 [ 488.811141] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 488.817112] LustreError: 15070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 489.118674] Lustre: server umount lustre-MDT0001 complete [ 494.303690] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 13:29:45 (1788802185) [ 511.770664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 512.197659] LustreError: 19918:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 512.276820] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 516.719710] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 521.722133] LustreError: 19918:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 521.748097] LustreError: 19918:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 525.260536] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 530.440259] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 537.402668] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 543.321148] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 546.049216] Lustre: Failing over lustre-MDT0000 [ 546.064523] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 546.077698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 546.082561] Lustre: Skipped 2 previous similar messages [ 546.101489] Lustre: Skipped 3 previous similar messages [ 546.515891] Lustre: server umount lustre-MDT0000 complete [ 550.187928] LustreError: 19914:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 550.208736] LustreError: 19914:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 565.003172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 565.093624] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 570.343203] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 570.367331] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 570.390193] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 570.767419] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 580.667549] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 585.705831] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 585.709889] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 585.716813] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 585.728169] Lustre: Skipped 1 previous similar message [ 591.304369] Lustre: server umount lustre-MDT0000 complete [ 594.926172] LustreError: 19900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788802286 with bad export cookie 16019378834491442890 [ 594.927676] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 594.934694] LustreError: 19900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 595.436708] Lustre: server umount lustre-MDT0001 complete [ 604.487041] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 604.959808] LustreError: 23255:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 605.000843] LustreError: 23255:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 605.067994] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 605.076935] Lustre: Skipped 2 previous similar messages [ 609.434347] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 618.319791] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 624.580939] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 632.540939] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 638.001574] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 640.230430] Lustre: Failing over lustre-MDT0000 [ 640.485489] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 640.590864] Lustre: server umount lustre-MDT0000 complete [ 658.921609] Lustre: 3646:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788802334/real 1788802334] req@ffff8f04823ce300 x1875694611263360/t0(0) o400->MGC192.168.201.104@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788802350 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 658.945603] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 659.876885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 669.166515] LustreError: 23256:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 669.190300] LustreError: 23256:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 18 previous similar messages [ 669.551147] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 674.387916] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 674.789816] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 674.795875] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 674.808116] Lustre: Skipped 1 previous similar message [ 674.828530] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 683.559707] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 688.816729] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 688.820350] Lustre: Skipped 4 previous similar messages [ 694.101567] Lustre: server umount lustre-MDT0000 complete [ 698.131990] LustreError: 23235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788802389 with bad export cookie 16019378834491443828 [ 698.145112] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 698.152420] LustreError: 23235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 698.579989] Lustre: server umount lustre-MDT0001 complete [ 707.833383] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 13:33:18 (1788802398) [ 724.642758] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [ 734.622356] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 782.313619] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 794.732769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 795.085875] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 795.133993] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 795.215623] Lustre: lustre-MDT0000: new disk, initializing [ 795.295686] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 795.305480] Lustre: Skipped 2 previous similar messages [ 795.319874] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 800.502960] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 811.638611] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 811.769883] Lustre: 29635:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 811.973721] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 816.479296] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 825.969208] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 833.654243] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 842.241733] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 842.700411] Lustre: 31602:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 843.930143] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 844.019501] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 844.185137] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 844.205529] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 850.794435] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 861.122602] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 866.304104] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 885.219399] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 885.229167] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 885.233416] Lustre: Skipped 4 previous similar messages [ 885.240608] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 885.244816] Lustre: Skipped 1 previous similar message [ 891.605981] Lustre: server umount lustre-OST0000 complete [ 899.041094] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 902.140856] Lustre: server umount lustre-MDT0000 complete [ 905.696239] LustreError: 29647:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 905.731235] LustreError: 29647:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 907.252015] LustreError: 29626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788802598 with bad export cookie 16019378834491444234 [ 907.257676] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 907.276893] LustreError: 29626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 908.535156] Lustre: server umount lustre-MDT0001 complete [ 923.639429] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 926.941407] Key type lgssc unregistered [ 927.214417] LNet: 33581:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 927.225833] LNetError: 33581:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 928.294831] LNet: Removed LNI 192.168.201.104@tcp [ 929.209294] Key type .llcrypt unregistered [ 929.212430] Key type ._llcrypt unregistered [ 958.324256] Key type ._llcrypt registered [ 958.325885] Key type .llcrypt registered [ 958.503721] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 959.797493] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 959.916555] alg: No test for adler32 (adler32-zlib) [ 960.932530] Lustre: Lustre: Build Version: 2.17.58_39_gb44c962 [ 961.204487] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 963.007688] Key type lgssc registered [ 964.537211] Lustre: Echo OBD driver; http://www.lustre.org/ [ 973.820874] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 973.871728] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 975.496288] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 980.508083] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 985.063105] LustreError: 34991:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 985.068642] LustreError: 34991:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 989.596274] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 989.910116] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 994.892575] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 1003.483796] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1009.218172] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1017.079603] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1017.466241] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1020.620921] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1024.474550] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 1032.744930] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1034.944263] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1050.350888] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 13:39:01 (1788802741) [ 1052.574871] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1054.492263] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1061.830327] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 13:39:12 (1788802752) [ 1072.360330] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 13:39:22 (1788802762) [ 1073.257212] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1081.891586] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1081.899977] Lustre: Skipped 2 previous similar messages [ 1084.094803] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 13:39:34 (1788802774) [ 1086.380093] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1086.383471] Lustre: Skipped 2 previous similar messages [ 1094.442248] Lustre: Setting parameter lustre-OST0000-osc.osc.max_pages_per_rpc=1024 in log lustre-client [ 1094.449952] Lustre: Skipped 1 previous similar message [ 1111.231884] Lustre: Disabling parameter general.osc.*.max_pages_per_rpc= in log params [ 1111.237294] Lustre: Skipped 8 previous similar messages [ 1113.228558] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 13:40:03 (1788802803) [ 1116.644665] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1120.346323] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1130.708931] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 13:40:20 (1788802820) [ 1150.142246] Lustre: Setting parameter general.jobid_name=jobid_name=jobid_name=%e.root in log params [ 1150.146637] Lustre: Skipped 4 previous similar messages [ 1152.467772] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 13:40:42 (1788802842) [ 1168.352322] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1168.357321] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1168.361748] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1168.379425] Lustre: Skipped 1 previous similar message [ 1173.484051] LustreError: 36814:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1173.506797] LustreError: 36814:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1173.785935] Lustre: server umount lustre-OST0000 complete [ 1178.596441] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1178.614875] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1178.624374] Lustre: Skipped 2 previous similar messages [ 1183.591994] Lustre: server umount lustre-MDT0000 complete [ 1183.718197] LustreError: 34990:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1183.740035] LustreError: 34990:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1188.111117] LustreError: 34972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788802879 with bad export cookie 12250012014542986172 [ 1188.114824] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1188.118839] LustreError: 34972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1188.428793] Lustre: server umount lustre-MDT0001 complete [ 1201.019379] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 1204.200710] Key type lgssc unregistered [ 1204.453855] LNet: 42853:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1204.463037] LNetError: 42853:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1204.488830] LNet: Removed LNI 192.168.201.104@tcp [ 1205.276145] Key type .llcrypt unregistered [ 1205.280262] Key type ._llcrypt unregistered [ 1225.497291] Key type ._llcrypt registered [ 1225.499381] Key type .llcrypt registered [ 1225.590681] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [ 1241.112857] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 1242.194580] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1242.339803] alg: No test for adler32 (adler32-zlib) [ 1243.522365] Lustre: Lustre: Build Version: 2.17.58_39_gb44c962 [ 1243.764675] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 1245.495212] Key type lgssc registered [ 1246.265946] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1289.267962] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1289.343823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1290.949250] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1291.010747] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1291.271667] Lustre: lustre-MDT0000: new disk, initializing [ 1291.429546] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1291.471047] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1298.058053] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 1311.169083] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1311.314658] Lustre: 46005:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1311.399433] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1311.405620] Lustre: Skipped 1 previous similar message [ 1311.536104] Lustre: lustre-MDT0001: new disk, initializing [ 1311.667419] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1311.733602] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1311.745154] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1317.024946] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 1327.097658] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1334.129100] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1342.115867] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1342.346649] Lustre: lustre-OST0000: new disk, initializing [ 1342.350128] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1342.355618] Lustre: 47973:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1342.422690] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1350.054086] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 1350.204225] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1350.213435] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1350.468585] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1360.441640] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1366.113153] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1366.356594] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1371.329523] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1371.591584] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1376.224824] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1376.232282] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1376.245966] Lustre: Skipped 1 previous similar message [ 1376.249277] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1376.261917] Lustre: Skipped 1 previous similar message [ 1379.624215] Lustre: server umount lustre-OST0000 complete [ 1386.466389] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1386.496588] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1389.973762] Lustre: server umount lustre-MDT0000 complete [ 1394.215685] LustreError: 47972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788803085 with bad export cookie 11986832216775150256 [ 1394.222129] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1394.230942] LustreError: 47972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1394.843781] Lustre: server umount lustre-MDT0001 complete [ 1406.571853] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 13:44:57 (1788803097) [ 1413.147109] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1413.873608] LustreError: 50434:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1413.903984] LustreError: 50434:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 1414.032214] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1418.622516] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 1419.250985] LustreError: 50435:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1419.267700] LustreError: 50435:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1424.355576] LustreError: 50434:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1427.361124] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1427.803402] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1432.763177] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 1440.701827] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1446.139720] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1452.855941] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1453.211256] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1459.802244] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 1467.839629] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1475.658176] LustreError: 52962:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1476.559618] Lustre: Modifying parameter general.timeout=1 in log params [ 1478.444909] Lustre: Modifying parameter general.timeout=3 in log params [ 1478.452677] Lustre: Skipped 1 previous similar message [ 1481.005486] Lustre: Modifying parameter general.timeout=6 in log params [ 1481.009427] Lustre: Skipped 2 previous similar messages [ 1485.791628] Lustre: Modifying parameter general.timeout=11 in log params [ 1485.796823] Lustre: Skipped 4 previous similar messages [ 1493.794885] Lustre: Modifying parameter general.timeout=20 in log params [ 1493.798228] Lustre: Skipped 8 previous similar messages [ 1510.032741] Lustre: Modifying parameter general.timeout=38 in log params [ 1510.038516] Lustre: Skipped 17 previous similar messages [ 1542.641254] Lustre: Modifying parameter general.timeout=75 in log params [ 1542.645712] Lustre: Skipped 36 previous similar messages [ 1604.538266] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 13:48:15 (1788803295) [ 1607.360479] Lustre: 59900:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1608.262920] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 in log params [ 1608.269855] Lustre: Skipped 57 previous similar messages [ 1609.328968] Lustre: 60011:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1613.132843] Lustre: 60210:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1613.154928] Lustre: 60210:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1647.556144] Lustre: 61780:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1647.566720] Lustre: 61780:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1647.596104] LustreError: 61780:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1652.518828] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 13:49:03 (1788803343) [ 1668.577432] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1668.585136] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1668.596233] Lustre: Skipped 1 previous similar message [ 1668.602198] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1668.604969] Lustre: Skipped 1 previous similar message [ 1672.684202] Lustre: server umount lustre-MDT0000 complete [ 1675.239387] LustreError: 50434:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1675.248706] LustreError: 50434:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1676.449409] LustreError: 50415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788803367 with bad export cookie 11986832216775151138 [ 1676.452231] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1676.464408] LustreError: 50415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1676.719964] Lustre: server umount lustre-MDT0001 complete [ 1691.440783] Lustre: server umount lustre-OST0000 complete [ 1719.607123] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1720.052961] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1720.468312] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1724.972613] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 1733.223070] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1733.348860] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1733.381467] Lustre: 64314:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1739.393870] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 1747.301865] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1753.335230] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1759.664586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1759.878217] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1760.182811] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1760.185435] Lustre: Skipped 1 previous similar message [ 1766.738673] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 1776.924412] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1779.394676] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1779.404645] Lustre: Skipped 20 previous similar messages [ 1779.513945] Lustre: 66877:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1788.386436] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1788.389404] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1788.403225] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1788.412154] Lustre: Skipped 3 previous similar messages [ 1788.430456] Lustre: Skipped 4 previous similar messages [ 1793.420572] Lustre: server umount lustre-OST0000 complete [ 1798.636956] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1798.638345] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1798.658627] Lustre: Skipped 1 previous similar message [ 1798.674843] Lustre: Skipped 1 previous similar message [ 1804.317373] Lustre: server umount lustre-MDT0000 complete [ 1807.848524] LustreError: 64324:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1807.868595] LustreError: 64324:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1808.141644] LustreError: 64304:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788803499 with bad export cookie 11986832216775154162 [ 1808.144429] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1808.149660] LustreError: 64304:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1808.665668] Lustre: server umount lustre-MDT0001 complete [ 1818.381989] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 1820.825847] Key type lgssc unregistered [ 1821.125638] LNet: 68084:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1821.138893] LNetError: 68084:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1821.157941] LNet: Removed LNI 192.168.201.104@tcp [ 1821.899566] Key type .llcrypt unregistered [ 1821.906995] Key type ._llcrypt unregistered [ 1844.439184] Key type ._llcrypt registered [ 1844.442246] Key type .llcrypt registered [ 1844.553895] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 1845.766619] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1845.787819] alg: No test for adler32 (adler32-zlib) [ 1847.002842] Lustre: Lustre: Build Version: 2.17.58_39_gb44c962 [ 1847.273090] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 1848.943551] Key type lgssc registered [ 1850.227555] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1859.136737] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1859.171049] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1860.749543] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1865.502812] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 1870.307503] LustreError: 69492:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1870.318554] LustreError: 69492:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1873.456543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1874.139124] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1878.797229] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 1886.223638] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1891.372967] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1898.167829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1898.721686] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1905.991774] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 1914.904418] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1922.424285] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1935.337365] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1935.350667] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1935.371552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1935.858226] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1935.875105] Lustre: Skipped 1 previous similar message [ 1935.888677] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1935.892673] Lustre: Skipped 1 previous similar message [ 1937.214996] Lustre: server umount lustre-MDT0000 complete [ 1940.967088] LustreError: 69492:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1940.984345] LustreError: 69492:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1941.257914] LustreError: 69474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788803632 with bad export cookie 16595159326222129698 [ 1941.264196] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1941.271943] LustreError: 69474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1941.606200] Lustre: server umount lustre-MDT0001 complete [ 1955.079843] Lustre: server umount lustre-OST0000 complete [ 1983.306133] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1983.733359] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1983.774541] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1984.249405] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1988.321338] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 1998.785269] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1998.967765] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1999.031878] Lustre: 74366:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1999.861056] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2007.431238] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2016.029841] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2024.375320] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2032.826043] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2033.090815] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2033.756840] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2041.648944] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2050.131887] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2051.708110] Lustre: 76944:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2053.710427] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2064.864705] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2064.867935] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2064.879606] LustreError: Skipped 1 previous similar message [ 2064.880540] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2064.915859] Lustre: Skipped 1 previous similar message [ 2066.527898] Lustre: server umount lustre-OST0000 complete [ 2071.010482] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2071.025310] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2071.056227] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2071.073096] Lustre: Skipped 1 previous similar message [ 2075.110229] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2075.111166] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2075.130069] Lustre: Skipped 1 previous similar message [ 2085.344284] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2085.346159] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 4. Is it stuck? [ 2085.354477] Lustre: Skipped 3 previous similar messages [ 2085.560696] Lustre: server umount lustre-MDT0000 complete [ 2089.400192] LustreError: 74359:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788803780 with bad export cookie 16595159326222130545 [ 2089.403463] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2089.422027] LustreError: 74359:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2090.136100] Lustre: server umount lustre-MDT0001 complete [ 2102.275366] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 2105.229237] Key type lgssc unregistered [ 2105.570172] LNet: 78159:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2105.575044] LNetError: 78159:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2106.599247] LNet: Removed LNI 192.168.201.104@tcp [ 2107.327269] Key type .llcrypt unregistered [ 2107.329399] Key type ._llcrypt unregistered [ 2131.866989] Key type ._llcrypt registered [ 2131.871663] Key type .llcrypt registered [ 2132.009779] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 2132.990502] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2133.189225] alg: No test for adler32 (adler32-zlib) [ 2134.246461] Lustre: Lustre: Build Version: 2.17.58_39_gb44c962 [ 2134.439070] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 2136.087276] Key type lgssc registered [ 2137.236690] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2145.273735] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2145.307766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2146.975948] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2152.813826] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2157.541733] LustreError: 79569:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2157.559279] LustreError: 79569:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2161.868155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2162.605361] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2167.948820] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2176.582945] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2182.657663] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2190.708735] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2191.340699] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2198.775319] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2207.702606] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2213.663324] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2214.717910] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2216.646372] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2216.652344] Lustre: Skipped 1 previous similar message [ 2219.600927] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2219.613704] Lustre: Skipped 2 previous similar messages [ 2224.527174] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2224.529941] Lustre: Skipped 4 previous similar messages [ 2233.139238] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2233.145888] Lustre: Skipped 8 previous similar messages [ 2249.724825] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2249.728557] Lustre: Skipped 17 previous similar messages [ 2282.532601] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2282.545684] Lustre: Skipped 34 previous similar messages [ 2316.203631] Lustre: 87670:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2319.720646] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 14:00:10 (1788804010) [ 2321.737146] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2324.288502] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 14:00:14 (1788804014) [ 2329.579924] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2329.582363] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2329.612384] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2329.613740] Lustre: Skipped 1 previous similar message [ 2329.645291] Lustre: Skipped 1 previous similar message [ 2334.688587] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2334.693631] Lustre: Skipped 1 previous similar message [ 2339.834780] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2339.840857] Lustre: Skipped 1 previous similar message [ 2343.394468] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2343.576059] Lustre: server umount lustre-OST0000 complete [ 2350.081910] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2350.084073] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2350.092606] Lustre: Skipped 1 previous similar message [ 2350.109411] Lustre: Skipped 1 previous similar message [ 2355.170078] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2355.197658] Lustre: Skipped 1 previous similar message [ 2362.335185] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2362.600760] Lustre: server umount lustre-MDT0000 complete [ 2365.412430] LustreError: 80300:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2365.429948] LustreError: 80300:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2366.672543] LustreError: 79551:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788804057 with bad export cookie 15224154345713862359 [ 2366.680634] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2366.683670] LustreError: 79551:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2367.372097] Lustre: server umount lustre-MDT0001 complete [ 2381.923853] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 2384.579550] Key type lgssc unregistered [ 2384.875248] LNet: 88934:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2384.890656] LNetError: 88934:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2384.912284] LNet: Removed LNI 192.168.201.104@tcp [ 2385.680444] Key type .llcrypt unregistered [ 2385.682534] Key type ._llcrypt unregistered [ 2389.694455] Key type ._llcrypt registered [ 2389.696649] Key type .llcrypt registered [ 2389.860287] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2394.836255] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules [ 2396.151884] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2396.190472] alg: No test for adler32 (adler32-zlib) [ 2397.367494] Lustre: Lustre: Build Version: 2.17.58_39_gb44c962 [ 2397.577355] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 2399.311258] Key type lgssc registered [ 2401.169297] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2402.752370] LustreError: 90157:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2422.526612] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 2424.888771] LustreError: 90157:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2432.767305] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2432.795498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2434.558152] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2440.112833] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2443.748387] LustreError: 91173:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2443.769140] LustreError: 91173:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2448.877601] LustreError: 91174:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2449.565637] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 14:02:20 (1788804140) [ 2453.994300] LustreError: 91173:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2454.858134] LustreError: 91205:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2454.871580] LustreError: 91205:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2454.889040] LustreError: 91205:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 21, retries 0, failed: rc = -5 [ 2455.383681] Lustre: server umount lustre-MDT0000 complete [ 2471.976433] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 2475.170588] Key type lgssc unregistered [ 2475.527340] LNet: 92699:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2475.532756] LNetError: 92699:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2475.556595] LNet: Removed LNI 192.168.201.104@tcp [ 2476.371925] Key type .llcrypt unregistered [ 2476.374699] Key type ._llcrypt unregistered [ 2496.776241] Key type ._llcrypt registered [ 2496.780598] Key type .llcrypt registered [ 2497.050622] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 2498.303197] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2498.359358] alg: No test for adler32 (adler32-zlib) [ 2499.666673] Lustre: Lustre: Build Version: 2.17.58_39_gb44c962 [ 2500.040864] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 2501.807205] Key type lgssc registered [ 2503.389771] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2513.154015] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2513.173817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2514.699365] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2520.292182] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2524.155941] LustreError: 93914:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2524.184730] LustreError: 93914:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2529.253432] LustreError: 93913:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2530.322138] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2531.306275] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2536.582747] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2543.864422] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2549.878845] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2557.209449] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2557.834798] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2566.957213] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2576.473512] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2627.591356] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 14:05:18 (1788804318) [ 2629.572661] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2631.429744] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 14:05:22 (1788804322) [ 2637.797984] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2637.813067] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2637.834602] Lustre: Skipped 2 previous similar messages [ 2637.863723] Lustre: Skipped 2 previous similar messages [ 2642.058105] Lustre: server umount lustre-MDT0000 complete [ 2642.921940] LustreError: 93914:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2642.950879] LustreError: 93914:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2646.529270] LustreError: 93894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788804337 with bad export cookie 1755088928398700369 [ 2646.530675] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2646.543613] LustreError: 93894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2646.923118] Lustre: server umount lustre-MDT0001 complete [ 2662.017593] Lustre: server umount lustre-OST0000 complete [ 2674.740408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2675.372087] LustreError: 98210:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2675.648968] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2680.800817] LustreError: 98211:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2681.165679] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2690.019039] LustreError: 98210:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2690.030627] LustreError: 98210:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2690.487931] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2690.977493] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2696.906307] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2706.023847] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2714.253421] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2731.279410] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2731.624498] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2731.634582] LustreError: 98199:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2731.678263] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2731.693384] LustreError: 100295:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -98 [ 2731.700442] LustreError: 100295:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 2731.709499] LustreError: 100295:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2731.793717] Lustre: server umount lustre-OST0000 complete [ 2731.797502] LustreError: 100295:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2741.303070] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2741.563444] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2741.570606] LustreError: 98199:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2741.599802] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2741.610578] LustreError: 100758:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -98 [ 2741.619116] LustreError: 100758:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 2741.627710] LustreError: 100758:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2741.750488] Lustre: server umount lustre-OST0000 complete [ 2741.756395] LustreError: 100758:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2752.981927] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2753.231915] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2753.234610] Lustre: Found index 0 for lustre-OST0000, updating log [ 2753.244987] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2753.387834] Lustre: lustre-OST0000: new disk, initializing [ 2753.395790] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2753.411908] Lustre: 101296:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2753.766158] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2761.080262] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2765.734951] Lustre: Failing over lustre-OST0000 [ 2766.106267] Lustre: server umount lustre-OST0000 complete [ 2772.960049] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2772.969624] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2772.992108] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2775.010194] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2775.012382] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2776.428663] Lustre: server umount lustre-MDT0000 complete [ 2780.129619] LustreError: 98210:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2780.153875] LustreError: 98210:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2780.213933] LustreError: 98190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788804471 with bad export cookie 1755088928398703113 [ 2780.224719] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2780.228109] LustreError: 98190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2790.023738] Lustre: server umount lustre-MDT0001 complete [ 2799.695029] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 14:08:10 (1788804490) [ 2818.627949] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 2830.874220] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2831.309539] LustreError: 103645:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2831.419388] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2836.429762] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2845.562920] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2851.625681] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2855.948263] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2863.687700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2864.869389] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2864.884597] Lustre: Skipped 1 previous similar message [ 2864.899306] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2871.013753] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2871.067321] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2871.069092] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 2872.846475] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2881.896673] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2881.965440] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2882.060796] Lustre: lustre-OST0001: new disk, initializing [ 2882.064400] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2882.067461] Lustre: 105964:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2887.836574] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2887.854848] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2887.988358] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2889.135124] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2897.860614] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2903.623085] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2911.891235] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.4@tcp (stopping) [ 2911.901783] Lustre: Skipped 1 previous similar message [ 2913.251785] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2913.273060] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2915.523350] Lustre: server umount lustre-MDT0001 complete [ 2916.965829] LustreError: 103641:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2917.005232] LustreError: 103641:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2927.763599] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2927.893846] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2927.911177] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2928.024316] Lustre: 103633:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2928.969631] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2928.986654] Lustre: Skipped 1 previous similar message [ 2929.573288] LustreError: 107500:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2929.584645] LustreError: 107500:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2929.591640] LustreError: 103637:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2929.781285] LustreError: 107508:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.104@tcp: cfg command failed: rc = -17 [ 2929.810294] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2929.810294] [ 2933.291811] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2934.281488] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2944.799523] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 14:10:35 (1788804635) [ 2966.813761] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 2988.813413] Lustre: 109581:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3015.596638] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3021.394917] Lustre: 110717:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3118.049322] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3118.064904] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3118.091512] Lustre: Skipped 2 previous similar messages [ 3118.104037] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3118.112692] Lustre: Skipped 3 previous similar messages [ 3120.852766] Lustre: server umount lustre-MDT0000 complete [ 3123.681753] LustreError: 106946:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3123.710296] LustreError: 106946:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [ 3126.108430] LustreError: 106682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788804817 with bad export cookie 1755088928398704114 [ 3126.114361] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3126.126032] LustreError: 106682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3126.738857] Lustre: server umount lustre-MDT0001 complete [ 3142.656450] Lustre: server umount lustre-OST0000 complete [ 3145.375112] Lustre: 93222:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788804820/real 1788804820] req@ffff8f05b1103800 x1875697143754368/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788804836 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3145.409383] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3145.419958] Lustre: Skipped 3 previous similar messages [ 3146.951850] Lustre: server umount lustre-OST0001 complete [ 3153.395366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3164.854799] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3174.871806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3186.175038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3218.636403] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 3230.598348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3230.655779] Lustre: lustre-MDT0000: reset Object Index mappings [ 3231.527677] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3237.169813] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3246.653678] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3246.681868] Lustre: lustre-MDT0001: reset Object Index mappings [ 3253.311246] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3256.653935] Lustre: 115851:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3263.794038] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3272.195353] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:232 to 0x280000400:257) [ 3272.236171] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3276.348989] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:349 to 0x280000401:386) [ 3282.496535] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3288.582220] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 3288.732687] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 3290.737808] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3299.720353] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3303.313548] Lustre: 117725:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3353.612579] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 14:17:24 (1788805044) [ 3380.708353] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3380.735046] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3380.737843] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3380.751218] Lustre: Skipped 1 previous similar message [ 3380.762443] Lustre: Skipped 5 previous similar messages [ 3385.835239] LustreError: 115439:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3385.864155] LustreError: 115439:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 12 previous similar messages [ 3385.957104] Lustre: server umount lustre-MDT0000 complete [ 3389.965812] LustreError: 115454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788805081 with bad export cookie 1755088928398751021 [ 3389.972582] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3389.986235] LustreError: 115454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3396.602569] Lustre: server umount lustre-MDT0001 complete [ 3407.784404] Lustre: server umount lustre-OST0000 complete [ 3418.354669] Lustre: server umount lustre-OST0001 complete [ 3426.194176] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [ 3437.892672] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 3480.104084] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3480.437677] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3480.459826] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3480.551162] Lustre: lustre-MDT0000: new disk, initializing [ 3480.668823] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3485.635971] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3494.353288] Lustre: server umount lustre-MDT0000 complete [ 3504.540506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3505.097396] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3505.114937] Lustre: 122006:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3505.126585] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3505.261569] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3505.265183] Lustre: Skipped 4 previous similar messages [ 3510.354919] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3515.113428] Lustre: server umount lustre-MDT0000 complete [ 3523.659979] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 14:20:14 (1788805214) [ 3525.256982] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3527.288385] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 14:20:18 (1788805218) [ 3547.604919] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 3559.180527] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3564.265767] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3573.469439] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3573.657138] Lustre: 124060:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3573.721748] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3573.728063] Lustre: Skipped 1 previous similar message [ 3573.829661] Lustre: lustre-MDT0001: new disk, initializing [ 3573.976542] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3573.985111] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3579.076863] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3583.845824] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3591.011825] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3591.339326] Lustre: lustre-OST0000: new disk, initializing [ 3591.343573] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3591.352517] Lustre: 125636:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3592.580495] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3592.616806] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3592.772311] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3599.005733] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3611.515273] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3611.702416] Lustre: lustre-OST0001: new disk, initializing [ 3611.706387] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3611.716874] Lustre: 126507:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3613.027841] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3613.048632] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3613.213739] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3618.754559] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3628.305914] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3632.318665] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3978.974429] Lustre: 132913:0:(osd_internal.h:1458:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3978.980617] Lustre: 132913:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3978.987028] Lustre: 132913:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3978.990027] Lustre: 132913:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3978.998387] Lustre: 132913:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3979.003804] Lustre: 132913:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4213.275314] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 14:31:43 (1788805903) [ 4229.600837] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4229.608304] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4229.620197] Lustre: Skipped 4 previous similar messages [ 4229.630695] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4229.639783] Lustre: Skipped 5 previous similar messages [ 4232.686670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4232.699466] Lustre: Skipped 1 previous similar message [ 4235.145362] Lustre: server umount lustre-MDT0000 complete [ 4237.808500] LustreError: 125631:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4237.822284] LustreError: 125631:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 4238.503511] LustreError: 132961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788805929 with bad export cookie 1755088928398818494 [ 4238.505487] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4238.514262] LustreError: 132961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4242.924992] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4242.929937] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4242.938690] Lustre: Skipped 4 previous similar messages [ 4242.956523] Lustre: Skipped 3 previous similar messages [ 4244.981105] Lustre: server umount lustre-MDT0001 complete [ 4263.391336] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4263.571862] Lustre: server umount lustre-OST0000 complete [ 4281.311733] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4281.744500] Lustre: server umount lustre-OST0001 complete [ 4288.615720] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [ 4298.778595] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 4331.549579] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4331.848216] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4331.884164] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4331.951278] Lustre: lustre-MDT0000: new disk, initializing [ 4332.058732] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4332.065294] Lustre: Skipped 4 previous similar messages [ 4332.078640] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4335.474285] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 4344.177982] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4344.329870] Lustre: 135961:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4344.373062] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4344.377295] Lustre: Skipped 1 previous similar message [ 4344.426538] Lustre: lustre-MDT0001: new disk, initializing [ 4344.513205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4344.517992] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4347.935779] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 4355.592995] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4359.186049] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4363.608859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4363.824802] Lustre: lustre-OST0000: new disk, initializing [ 4363.833151] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4363.839273] Lustre: 137926:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4365.344299] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4365.358276] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4365.498556] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4368.490647] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 4376.604833] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4384.371354] Lustre: lustre-MDD0000: changelog on [ 4386.640782] Lustre: lustre-MDD0001: changelog on [ 5636.009602] Lustre: lustre-MDD0001: changelog off [ 5637.742614] Lustre: 139381:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 5638.469867] Lustre: lustre-MDD0000: changelog off [ 5645.794351] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5645.808062] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5645.828211] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5645.833180] Lustre: Skipped 3 previous similar messages [ 5649.305985] Lustre: server umount lustre-OST0000 complete [ 5655.522388] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5655.531942] LustreError: Skipped 1 previous similar message [ 5655.542144] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5655.551269] Lustre: Skipped 1 previous similar message [ 5655.554739] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5661.152651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5661.156849] Lustre: Skipped 3 previous similar messages [ 5666.271100] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5666.276114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5666.288315] Lustre: Skipped 1 previous similar message [ 5666.534912] Lustre: server umount lustre-MDT0000 complete [ 5669.381394] LustreError: 135954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788807360 with bad export cookie 1755088928399634043 [ 5669.388255] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5669.389975] LustreError: 135954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5669.867232] Lustre: server umount lustre-MDT0001 complete [ 5678.657474] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 5680.738540] Key type lgssc unregistered [ 5681.008360] LNet: 146982:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5681.017353] LNetError: 146982:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5682.089651] LNet: Removed LNI 192.168.201.104@tcp [ 5682.938971] Key type .llcrypt unregistered [ 5682.942741] Key type ._llcrypt unregistered [ 5700.895149] Key type ._llcrypt registered [ 5700.903293] Key type .llcrypt registered [ 5701.061273] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [ 5715.484424] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 5716.381642] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5716.438746] alg: No test for adler32 (adler32-zlib) [ 5717.468633] Lustre: Lustre: Build Version: 2.17.58_39_gb44c962 [ 5717.681487] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 5719.383482] Key type lgssc registered [ 5720.556327] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5753.706351] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5753.734448] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5754.986995] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5755.009837] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5755.083592] Lustre: lustre-MDT0000: new disk, initializing [ 5755.145579] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5755.159443] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5759.122318] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 5767.605509] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5767.690413] Lustre: 150177:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5767.740877] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5767.744831] Lustre: Skipped 1 previous similar message [ 5767.812905] Lustre: lustre-MDT0001: new disk, initializing [ 5767.918549] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5767.964853] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5767.982186] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5771.596246] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 5779.275558] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5783.326667] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5788.331184] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5788.547941] Lustre: lustre-OST0000: new disk, initializing [ 5788.550850] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5788.555762] Lustre: 152145:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5788.626500] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5793.558328] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 5798.444625] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5798.455936] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5798.530755] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5801.235460] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5806.828583] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_module obdecho/obdecho [ 5808.370748] LustreError: 153147:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 5808.375811] LustreError: 153147:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 5809.805409] LustreError: 153243:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 5818.852625] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5818.855697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5818.883986] Lustre: Skipped 2 previous similar messages [ 5818.889815] Lustre: Skipped 1 previous similar message [ 5823.972334] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5823.982468] Lustre: Skipped 3 previous similar messages [ 5824.696765] Lustre: server umount lustre-MDT0000 complete [ 5828.635251] LustreError: 150971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788807519 with bad export cookie 7917053116847722783 [ 5828.636765] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5828.646102] LustreError: 150971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5828.908210] Lustre: server umount lustre-MDT0001 complete [ 5843.245380] Lustre: server umount lustre-OST0000 complete [ 5862.767861] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [ 5875.235651] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 5913.170547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5913.475414] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5913.525833] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5913.606398] Lustre: lustre-MDT0000: new disk, initializing [ 5913.746809] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5913.761204] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5919.034722] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 5930.428452] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5930.589775] Lustre: 156963:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5930.649724] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5930.659495] Lustre: Skipped 1 previous similar message [ 5930.755282] Lustre: lustre-MDT0001: new disk, initializing [ 5930.883181] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5930.989602] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5931.012179] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5935.794887] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 5946.404926] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5953.236648] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5959.778815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5960.230942] Lustre: lustre-OST0000: new disk, initializing [ 5960.236349] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5960.244106] Lustre: 158930:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5960.391910] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5965.913192] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5965.948522] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5966.117478] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5967.331801] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 5978.043143] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5983.262957] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5983.443077] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5988.736272] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5988.953477] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5991.904987] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5991.915239] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5991.936295] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5991.944764] Lustre: Skipped 1 previous similar message [ 5997.026099] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5997.043626] Lustre: Skipped 1 previous similar message [ 5997.689444] Lustre: server umount lustre-OST0000 complete [ 6002.145891] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6002.150868] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6002.153803] Lustre: Skipped 2 previous similar messages [ 6008.070243] Lustre: server umount lustre-MDT0000 complete [ 6011.266886] LustreError: 157757:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788807702 with bad export cookie 7917053116847723749 [ 6011.269867] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6011.274907] LustreError: 157757:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6011.586442] Lustre: server umount lustre-MDT0001 complete [ 6015.737105] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 15:01:46 (1788807706) [ 6017.625391] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6019.545992] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 15:01:50 (1788807710) [ 6027.716615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6028.423555] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6033.580382] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 6033.890207] LustreError: 161355:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6033.919608] LustreError: 161355:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6038.060825] LustreError: 161356:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6043.114543] LustreError: 161355:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6043.989850] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6050.269048] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 6058.188268] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6064.467177] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6072.083126] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6072.472772] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6072.478090] Lustre: Skipped 1 previous similar message [ 6079.549095] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 6088.560640] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6094.318355] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6094.325658] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6094.332982] Lustre: Skipped 2 previous similar messages [ 6094.344842] Lustre: Skipped 4 previous similar messages [ 6094.711476] Lustre: server umount lustre-MDT0000 complete [ 6098.636124] LustreError: 161336:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788807789 with bad export cookie 7917053116847724631 [ 6098.643365] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6098.646821] LustreError: 161336:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6099.430110] LustreError: 161351:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6099.434966] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6103.456402] LustreError: 161349:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6103.481674] LustreError: 161349:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6105.188895] Lustre: server umount lustre-MDT0001 complete [ 6111.532158] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6125.486405] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6125.753727] LustreError: 165196:0:(llog.c:1655:llog_backup()) MGC192.168.201.104@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6125.761766] Lustre: 165196:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.104@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6128.096712] LustreError: 165201:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6128.124239] LustreError: 165201:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6146.635984] LustreError: 165201:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6146.668294] LustreError: 165201:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 14 previous similar messages [ 6166.090243] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.4@tcp (not set up) [ 6166.093694] Lustre: Skipped 5 previous similar messages [ 6178.785666] LustreError: 165201:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6178.798506] LustreError: 165201:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 25 previous similar messages [ 6186.463316] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6186.466596] Lustre: 165196:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.104@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6227.451861] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6231.220265] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 6232.556752] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6239.234841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6242.789133] LustreError: 163187:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6242.812851] LustreError: 163187:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 42 previous similar messages [ 6247.508769] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 6248.428725] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6254.246938] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6255.621596] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6260.885073] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6262.396336] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6278.699785] Lustre: server umount lustre-MDT0000 complete [ 6279.141976] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6279.151675] LustreError: Skipped 1 previous similar message [ 6279.155762] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6279.164775] Lustre: Skipped 1 previous similar message [ 6282.090179] LustreError: 161337:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788807973 with bad export cookie 7917053116847731085 [ 6282.093412] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6282.100935] LustreError: 161337:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6282.557369] Lustre: server umount lustre-MDT0001 complete [ 6296.803724] Lustre: server umount lustre-OST0000 complete [ 6315.511722] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [ 6324.057857] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 6357.087358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6357.294188] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6357.314431] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6357.374431] Lustre: lustre-MDT0000: new disk, initializing [ 6357.429397] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6357.436224] Lustre: Skipped 1 previous similar message [ 6357.449543] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6361.529812] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 6370.799106] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6370.901169] Lustre: 170296:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6371.084935] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6374.707364] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 6382.293805] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6386.883903] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6393.000476] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6393.217675] Lustre: lustre-OST0000: new disk, initializing [ 6393.220443] Lustre: Skipped 1 previous similar message [ 6393.223987] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6393.231892] Lustre: Skipped 2 previous similar messages [ 6393.236352] Lustre: 172263:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6398.524835] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6398.537474] Lustre: Skipped 1 previous similar message [ 6398.545089] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6398.647245] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6398.748627] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 6406.942275] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6410.912594] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6411.086635] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6414.889355] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6415.043175] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6419.424691] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6419.425530] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6419.431785] LustreError: Skipped 1 previous similar message [ 6419.434929] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6419.446925] Lustre: Skipped 2 previous similar messages [ 6431.199504] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6431.360663] Lustre: server umount lustre-OST0000 complete [ 6437.860096] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6441.362398] Lustre: server umount lustre-MDT0000 complete [ 6444.811500] LustreError: 171101:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788808136 with bad export cookie 7917053116847747920 [ 6444.813299] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6444.827866] LustreError: 171101:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6445.147790] Lustre: server umount lustre-MDT0001 complete [ 6453.522799] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 15:09:04 (1788808144) [ 6460.103213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6460.697601] LustreError: 174674:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6460.707148] LustreError: 174674:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 6465.097503] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 6472.578036] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6476.537554] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 6482.855303] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6487.434448] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6492.852636] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6498.309741] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 6504.295685] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6519.057048] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 15:10:10 (1788808210) [ 6523.874698] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6523.884247] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6523.910118] Lustre: Skipped 2 previous similar messages [ 6523.914273] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6523.917123] Lustre: Skipped 8 previous similar messages [ 6529.092858] Lustre: server umount lustre-OST0000 complete [ 6534.624241] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6534.632906] LustreError: Skipped 1 previous similar message [ 6539.267806] Lustre: server umount lustre-MDT0000 complete [ 6542.481048] LustreError: 174656:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788808233 with bad export cookie 7917053116847748802 [ 6542.484763] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6542.495741] LustreError: 174656:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6542.957405] Lustre: server umount lustre-MDT0001 complete [ 6552.107399] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 6554.536897] Key type lgssc unregistered [ 6554.814175] LNet: 178399:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6554.819871] LNetError: 178399:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6555.876889] LNet: Removed LNI 192.168.201.104@tcp [ 6556.623848] Key type .llcrypt unregistered [ 6556.626198] Key type ._llcrypt unregistered [ 6571.980822] Key type ._llcrypt registered [ 6571.986783] Key type .llcrypt registered [ 6572.108462] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 6572.896555] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6572.911554] alg: No test for adler32 (adler32-zlib) [ 6573.940765] Lustre: Lustre: Build Version: 2.17.58_39_gb44c962 [ 6574.155342] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 6575.807224] Key type lgssc registered [ 6576.676449] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6583.412325] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6583.427713] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6598.943607] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6604.063499] LustreError: 179619:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.104@tcp: failed processing log, type 4: rc = -110 [ 6609.183465] LustreError: 179619:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 6609.192081] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 6609.195812] LustreError: MGC192.168.201.104@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 6609.203859] LustreError: 179619:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-OST0000: -22 [ 6609.210454] LustreError: 179619:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -22 [ 6609.215587] LustreError: 179619:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 6609.286372] Lustre: server umount lustre-OST0000 complete [ 6609.290449] LustreError: 179619:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6616.150959] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6617.477862] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6620.771117] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 6626.785881] LustreError: 180120:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6626.800769] LustreError: 180120:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6627.951777] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6628.264324] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6632.916754] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 6639.517636] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6644.656579] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6650.852687] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6651.016361] LustreError: 181933:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 6651.023752] Lustre: 181933:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.201.104@tcp: invalid local config log lustre-OST0000: rc = -22 [ 6651.164218] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6656.584329] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 6663.633994] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6667.747222] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6667.750374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6667.760283] Lustre: Skipped 2 previous similar messages [ 6667.767597] Lustre: Skipped 2 previous similar messages [ 6672.869421] LustreError: 181958:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6672.886798] LustreError: 181958:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6673.877722] Lustre: server umount lustre-MDT0000 complete [ 6677.156186] LustreError: 180099:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788808368 with bad export cookie 7428748570540214922 [ 6677.160523] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6677.176193] LustreError: 180099:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6677.444558] Lustre: server umount lustre-MDT0001 complete [ 6691.715121] Lustre: server umount lustre-OST0000 complete [ 6700.212415] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [ 6709.914764] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 6741.143495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6741.339090] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6741.363860] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6741.446422] Lustre: lustre-MDT0000: new disk, initializing [ 6741.538503] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6741.559444] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6745.398383] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 6754.779542] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6754.906521] Lustre: 185457:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6754.970796] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6754.974605] Lustre: Skipped 1 previous similar message [ 6755.038793] Lustre: lustre-MDT0001: new disk, initializing [ 6755.112993] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6755.150160] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6755.159814] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6759.068749] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 6767.101516] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6771.176951] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6776.051428] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6776.265371] Lustre: lustre-OST0000: new disk, initializing [ 6776.268824] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6776.274325] Lustre: 187422:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6776.363168] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6781.625704] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 6784.569140] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6784.576139] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6784.683470] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6789.096416] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6792.934754] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6793.156611] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6796.970484] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6797.176468] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6800.352903] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6800.356691] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6800.373693] LustreError: Skipped 1 previous similar message [ 6800.375713] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6800.393521] Lustre: Skipped 1 previous similar message [ 6805.476128] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6805.483787] Lustre: Skipped 2 previous similar messages [ 6810.601103] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6810.614734] Lustre: Skipped 1 previous similar message [ 6813.151162] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6813.375613] Lustre: server umount lustre-OST0000 complete [ 6820.834656] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6820.859862] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6820.872773] Lustre: Skipped 1 previous similar message [ 6823.193697] Lustre: server umount lustre-MDT0000 complete [ 6825.954032] LustreError: 188886:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6825.976625] LustreError: 188886:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6826.533500] LustreError: 187434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788808517 with bad export cookie 7428748570540215643 [ 6826.542549] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6826.547469] LustreError: 187434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6826.840458] Lustre: server umount lustre-MDT0001 complete [ 6833.402518] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 15:15:24 (1788808524) [ 6845.358354] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 6847.349945] Key type lgssc unregistered [ 6847.549951] LNet: 190366:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6847.554283] LNetError: 190366:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6847.570939] LNet: Removed LNI 192.168.201.104@tcp [ 6848.218699] Key type .llcrypt unregistered [ 6848.220825] Key type ._llcrypt unregistered [ 6861.870613] Key type ._llcrypt registered [ 6861.872703] Key type .llcrypt registered [ 6861.937364] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 6862.605681] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6862.660186] alg: No test for adler32 (adler32-zlib) [ 6863.603596] Lustre: Lustre: Build Version: 2.17.58_39_gb44c962 [ 6863.777714] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 6865.407221] Key type lgssc registered [ 6866.356793] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6872.137343] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6872.153805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6873.456797] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6876.876457] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 6882.787175] LustreError: 191557:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6882.806174] LustreError: 191557:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6883.507496] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6883.780773] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6887.568702] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 6893.377438] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6897.907167] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6903.981833] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6904.138855] Lustre: *** cfs_fail_loc=131e, val=0*** [ 6904.141722] LustreError: 193423:0:(llog.c:1696:llog_backup()) MGC192.168.201.104@tcp: failed to backup log lustre-OST0000: rc = -28 [ 6904.151574] Lustre: 193423:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.201.104@tcp: can't backup local config lustre-OST0000: rc = -28 [ 6904.251225] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6908.982632] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 6914.161087] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6919.652114] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6919.660415] Lustre: Skipped 2 previous similar messages [ 6919.663175] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6919.665048] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6919.672216] Lustre: Skipped 2 previous similar messages [ 6921.871616] Lustre: server umount lustre-MDT0000 complete [ 6924.608575] LustreError: 191539:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788808615 with bad export cookie 16982419911173195626 [ 6924.616921] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6924.786920] Lustre: server umount lustre-MDT0001 complete [ 6937.808419] Lustre: server umount lustre-OST0000 complete [ 6945.456788] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [ 6953.411505] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 6980.448951] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6980.637205] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6980.681949] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6980.754367] Lustre: lustre-MDT0000: new disk, initializing [ 6980.820755] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6980.836193] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6984.274442] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 6992.753036] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6992.890964] Lustre: 196947:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6992.943365] Lustre: *** cfs_fail_loc=131e, val=0*** [ 6992.945946] Lustre: Skipped 1 previous similar message [ 6992.949116] LustreError: 197728:0:(llog.c:1696:llog_backup()) MGC192.168.201.104@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 6992.955693] LustreError: 197728:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 6992.960562] Lustre: 197728:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.104@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 6992.981555] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6992.984256] Lustre: Skipped 1 previous similar message [ 6993.051192] Lustre: lustre-MDT0001: new disk, initializing [ 6993.139771] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6993.174074] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6993.186036] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6997.004259] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 7004.698567] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7009.066546] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7015.155656] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7015.368587] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7015.370015] Lustre: Skipped 1 previous similar message [ 7015.372476] LustreError: 198890:0:(llog.c:1696:llog_backup()) MGC192.168.201.104@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7015.380759] LustreError: 198890:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7015.388891] Lustre: 198890:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.104@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7015.394615] Lustre: 198890:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7015.411625] Lustre: lustre-OST0000: new disk, initializing [ 7015.415340] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7015.420733] Lustre: 198913:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7015.515531] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7021.009685] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 7022.621969] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7022.634234] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7022.696885] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7029.031063] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7033.075155] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7033.220915] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7036.670471] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7036.856176] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7043.045640] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7043.046830] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7043.057786] LustreError: Skipped 1 previous similar message [ 7043.058776] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7043.074441] Lustre: Skipped 1 previous similar message [ 7044.724917] Lustre: server umount lustre-OST0000 complete [ 7048.165284] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7048.169205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7048.179778] Lustre: Skipped 1 previous similar message [ 7048.193539] Lustre: Skipped 2 previous similar messages [ 7053.281202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7053.290870] Lustre: Skipped 1 previous similar message [ 7053.813143] Lustre: server umount lustre-MDT0000 complete [ 7056.955804] LustreError: 196937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788808748 with bad export cookie 16982419911173196340 [ 7056.956848] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7056.967796] LustreError: 196937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 7057.247547] Lustre: server umount lustre-MDT0001 complete [ 7064.617555] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 15:19:15 (1788808755) [ 7077.672524] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 7085.935612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7086.254966] LustreError: 202039:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7086.301723] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7090.459988] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 7091.682278] LustreError: 202040:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7095.788245] LustreError: 202039:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7096.575688] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7100.410548] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 7102.533909] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7107.690906] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7113.009450] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 7119.991771] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7120.125717] Lustre: lustre-OST0001: new disk, initializing [ 7120.129207] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7120.133596] Lustre: 204346:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7120.228758] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7120.235761] Lustre: Skipped 2 previous similar messages [ 7122.458439] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7122.468697] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7122.570114] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7125.533089] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 7134.343542] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7141.386403] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7142.732296] Lustre: DEBUG MARKER: ADD OST3 [ 7146.303685] loop: module loaded [ 7146.358959] loop0: detected capacity change from 0 to 204800000 [ 7146.538140] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7146.555099] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7146.583452] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7151.366974] loop0: detected capacity change from 0 to 204800000 [ 7152.889757] Lustre: DEBUG MARKER: STOP OST3 [ 7155.538648] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7155.649954] Lustre: lustre-OST0003: new disk, initializing [ 7155.652817] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 7155.657224] Lustre: 206275:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7157.138472] Lustre: Failing over lustre-OST0003 [ 7157.296659] Lustre: server umount lustre-OST0003 complete [ 7157.351781] LustreError: 204494:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7160.893130] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 7166.049448] LustreError: 203521:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.201.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7166.074470] LustreError: 203521:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 7169.276829] Lustre: DEBUG MARKER: START OST3 again [ 7175.090410] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7175.315199] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 7176.273546] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 1 client reconnects [ 7176.310935] Lustre: lustre-OST0003: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 7180.558687] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 7180.842383] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 7180.857972] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 7180.906774] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000401 [ 7198.310405] Lustre: Failing over lustre-OST0003 [ 7198.379979] Lustre: server umount lustre-OST0003 complete [ 7201.255578] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7201.261777] LustreError: 204494:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7201.262442] Lustre: Skipped 1 previous similar message [ 7201.263416] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7201.281159] LustreError: 204494:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 7214.566951] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7214.576795] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7214.595407] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7220.681311] Lustre: server umount lustre-MDT0000 complete [ 7223.491626] LustreError: 204344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788808914 with bad export cookie 16982419911173197215 [ 7223.502540] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7226.852965] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7226.856122] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7226.864849] Lustre: Skipped 4 previous similar messages [ 7226.874017] Lustre: Skipped 4 previous similar messages [ 7229.843203] Lustre: server umount lustre-MDT0001 complete [ 7233.397819] Lustre: server umount lustre-OST0000 complete [ 7236.451660] Lustre: server umount lustre-OST0001 complete [ 7242.106894] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [ 7248.904612] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 7277.797901] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7277.960157] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7277.980329] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7278.040374] Lustre: lustre-MDT0000: new disk, initializing [ 7278.097330] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7278.100131] Lustre: Skipped 2 previous similar messages [ 7278.115655] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7282.212760] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 7290.749215] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7290.896578] Lustre: 211799:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7291.186494] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7294.777723] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 7302.923932] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7307.408252] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7312.645738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7312.866517] Lustre: lustre-OST0000: new disk, initializing [ 7312.868694] Lustre: Skipped 1 previous similar message [ 7312.871085] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7312.874996] Lustre: Skipped 2 previous similar messages [ 7312.882621] Lustre: 213764:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7314.171313] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7314.193971] Lustre: Skipped 1 previous similar message [ 7314.203273] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7314.305131] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7318.884427] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 7326.593131] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7331.002926] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7331.192880] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7335.645965] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7335.815511] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7340.000545] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7340.001550] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7340.010974] LustreError: Skipped 1 previous similar message [ 7340.013169] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7340.013176] Lustre: Skipped 3 previous similar messages [ 7340.040128] Lustre: Skipped 1 previous similar message [ 7343.621633] Lustre: server umount lustre-OST0000 complete [ 7352.845716] Lustre: server umount lustre-MDT0000 complete [ 7353.313089] LustreError: 211809:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7353.323420] LustreError: 211809:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 15 previous similar messages [ 7356.278706] LustreError: 212592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788809047 with bad export cookie 16982419911173199504 [ 7356.280647] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7356.291403] LustreError: 212592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 7356.678387] Lustre: server umount lustre-MDT0001 complete [ 7359.794529] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 15:24:11 (1788809051) [ 7373.801883] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [ 7383.159598] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 7412.633358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7412.804367] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7412.825557] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7412.893994] Lustre: lustre-MDT0000: new disk, initializing [ 7412.961594] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7412.965816] Lustre: Skipped 2 previous similar messages [ 7412.978958] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7416.214896] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 7425.830677] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7425.937798] Lustre: 218368:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7426.164863] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7429.723257] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 7437.306541] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7442.881337] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7449.073738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7449.260359] Lustre: 220336:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7450.851973] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7454.687817] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 7462.246331] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7466.262829] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7466.446986] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7470.020411] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7470.204882] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7476.707429] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7476.711370] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7476.714274] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7476.714282] Lustre: Skipped 6 previous similar messages [ 7476.714818] LustreError: Skipped 2 previous similar messages [ 7476.734455] Lustre: Skipped 3 previous similar messages [ 7477.950844] Lustre: server umount lustre-OST0000 complete [ 7495.139471] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7495.266515] Lustre: server umount lustre-MDT0000 complete [ 7497.186807] LustreError: 219593:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7497.215871] LustreError: 219593:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7498.191245] LustreError: 218361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788809189 with bad export cookie 16982419911173200386 [ 7498.201305] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7498.205744] LustreError: 218361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7498.553851] Lustre: server umount lustre-MDT0001 complete [ 7505.686297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7509.934283] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 7516.930400] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7520.911921] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 7526.515851] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7530.850837] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7535.972538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7540.936312] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 7546.580862] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7562.208057] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7562.217855] Lustre: Skipped 2 previous similar messages [ 7562.219846] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7562.233601] Lustre: Skipped 8 previous similar messages [ 7566.999808] Lustre: server umount lustre-OST0000 complete [ 7576.602708] Lustre: server umount lustre-MDT0000 complete [ 7579.440219] LustreError: 222535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788809270 with bad export cookie 16982419911173201268 [ 7579.448893] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7579.453792] LustreError: 222535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7579.723701] Lustre: server umount lustre-MDT0001 complete [ 7684.191687] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 7686.389629] Key type lgssc unregistered [ 7686.646336] LNet: 226236:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7686.655636] LNetError: 226236:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7686.670868] LNet: Removed LNI 192.168.201.104@tcp [ 7687.300944] Key type .llcrypt unregistered [ 7687.302764] Key type ._llcrypt unregistered [ 7705.273352] Key type ._llcrypt registered [ 7705.274839] Key type .llcrypt registered [ 7705.338809] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [ 7715.473878] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 7716.207167] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7716.233130] alg: No test for adler32 (adler32-zlib) [ 7717.210381] Lustre: Lustre: Build Version: 2.17.58_39_gb44c962 [ 7717.368989] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 7718.994203] Key type lgssc registered [ 7719.826622] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7742.379237] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7742.395985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7743.581083] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7743.620496] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7743.679876] Lustre: lustre-MDT0000: new disk, initializing [ 7743.769823] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7743.785223] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7746.742058] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 7754.102948] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7754.187480] Lustre: 229560:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7754.229187] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7754.231766] Lustre: Skipped 1 previous similar message [ 7754.278893] Lustre: lustre-MDT0001: new disk, initializing [ 7754.339132] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7754.369112] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7754.378321] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7757.257872] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 7763.887639] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7767.461189] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7771.795849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7771.937858] Lustre: lustre-OST0000: new disk, initializing [ 7771.940270] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7771.943989] Lustre: 231526:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7771.994051] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7775.898840] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 7777.831879] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7777.839749] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7777.920865] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7782.240785] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7785.408684] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7785.539942] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7788.947672] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7789.149236] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7793.121192] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7793.132839] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7793.143424] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7793.633533] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7793.655448] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7796.700460] Lustre: server umount lustre-OST0000 complete [ 7800.292626] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7800.297280] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7800.305718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7803.872045] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7803.872511] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7805.629351] Lustre: server umount lustre-MDT0000 complete [ 7808.338059] LustreError: 230354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788809499 with bad export cookie 10588716164593677586 [ 7808.343514] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7808.347841] LustreError: 230354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7808.612598] Lustre: server umount lustre-MDT0001 complete [ 7821.355231] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 7828.365615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7828.814665] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7832.216623] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 7834.081033] LustreError: 234459:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7834.107500] LustreError: 234459:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7838.358061] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7838.597951] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7841.667511] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 7843.549750] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7847.859578] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7848.064741] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7851.737284] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 7857.603497] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7857.724729] Lustre: lustre-OST0001: new disk, initializing [ 7857.727724] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7857.733567] Lustre: 236763:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7859.027414] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7859.043723] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7859.121704] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7861.796598] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 7869.900834] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7875.111866] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7883.027299] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 15:32:54 (1788809574) [ 7884.768976] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7884.771424] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7884.777170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7884.779428] Lustre: Skipped 1 previous similar message [ 7890.506388] Lustre: server umount lustre-MDT0000 complete [ 7892.844528] LustreError: 234439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788809584 with bad export cookie 10588716164593678461 [ 7892.853857] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7892.854981] LustreError: 234439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7893.038543] Lustre: server umount lustre-MDT0001 complete [ 7905.783669] Lustre: server umount lustre-OST0000 complete [ 7918.532889] Lustre: server umount lustre-OST0001 complete [ 7922.705396] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [ 7928.875844] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 7951.661288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7951.763805] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7951.778703] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7951.816630] Lustre: lustre-MDT0000: new disk, initializing [ 7951.853674] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7951.857820] Lustre: Skipped 1 previous similar message [ 7951.870368] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7954.615523] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 7961.657642] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7961.740784] Lustre: 240703:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7961.775885] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7961.779112] Lustre: Skipped 1 previous similar message [ 7961.825517] Lustre: lustre-MDT0001: new disk, initializing [ 7961.917581] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7961.926653] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7964.564501] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 7970.830839] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7974.425523] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7978.850927] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7979.023248] Lustre: lustre-OST0000: new disk, initializing [ 7979.025376] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7979.029197] Lustre: 242670:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7980.687850] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7980.698195] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7980.779811] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7982.945804] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 7989.963284] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7992.970751] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7993.114840] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7996.097563] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7996.203703] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8001.509384] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8001.515652] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8001.525605] Lustre: Skipped 3 previous similar messages [ 8001.530604] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8001.534305] Lustre: Skipped 5 previous similar messages [ 8003.735322] Lustre: server umount lustre-OST0000 complete [ 8012.523564] Lustre: server umount lustre-MDT0000 complete [ 8015.032440] LustreError: 240696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788809706 with bad export cookie 10588716164593680603 [ 8015.033048] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8015.037490] LustreError: 240696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8015.270678] Lustre: server umount lustre-MDT0001 complete [ 8020.422625] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8020.686102] LustreError: 244883:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8020.750185] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8020.754791] Lustre: Skipped 2 previous similar messages [ 8023.393533] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8026.080760] LustreError: 244884:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8029.007947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8031.966723] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8036.623419] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8040.026618] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8043.989291] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8048.280180] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8052.822235] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8200.164819] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8200.173585] LustreError: Skipped 1 previous similar message [ 8200.180884] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8200.188784] Lustre: Skipped 3 previous similar messages [ 8200.191843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8200.197347] Lustre: Skipped 4 previous similar messages [ 8204.822641] Lustre: server umount lustre-OST0000 complete [ 8208.360678] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8208.368306] LustreError: Skipped 1 previous similar message [ 8213.476443] LustreError: 244884:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8213.489443] LustreError: 244884:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8213.538547] Lustre: server umount lustre-MDT0000 complete [ 8215.612404] LustreError: 244865:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788809906 with bad export cookie 10588716164593681478 [ 8215.616099] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8215.623932] LustreError: 244865:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8215.837451] Lustre: server umount lustre-MDT0001 complete [ 8317.116326] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 8318.933400] Key type lgssc unregistered [ 8319.116613] LNet: 248487:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8319.124735] LNetError: 248487:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8319.141752] LNet: Removed LNI 192.168.201.104@tcp [ 8319.777128] Key type .llcrypt unregistered [ 8319.778661] Key type ._llcrypt unregistered [ 8336.030663] Key type ._llcrypt registered [ 8336.032100] Key type .llcrypt registered [ 8336.101519] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [ 8346.172241] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 8346.794090] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8346.836753] alg: No test for adler32 (adler32-zlib) [ 8347.785813] Lustre: Lustre: Build Version: 2.17.58_39_gb44c962 [ 8347.942100] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 8349.559194] Key type lgssc registered [ 8350.196850] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8377.453222] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 8383.384241] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8383.397094] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8384.578548] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8384.611951] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8384.729128] Lustre: lustre-MDT0000: new disk, initializing [ 8384.807735] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8384.827065] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8387.565775] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8394.505679] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8394.582251] Lustre: 252532:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8394.625533] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8394.628510] Lustre: Skipped 1 previous similar message [ 8394.691120] Lustre: lustre-MDT0001: new disk, initializing [ 8394.747427] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8394.771327] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8394.776882] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8397.735680] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8401.108305] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8404.982939] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8405.130213] Lustre: lustre-OST0000: new disk, initializing [ 8405.134091] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8405.139391] Lustre: 254164:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8405.198664] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8408.152102] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8408.158715] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8408.223157] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8409.144399] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8415.964782] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8416.065463] Lustre: lustre-OST0001: new disk, initializing [ 8416.067374] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8416.071269] Lustre: 255034:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8416.112425] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8416.529386] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8416.541547] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8416.569786] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8419.804905] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8428.724102] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8438.201394] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8445.919903] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8445.925178] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8445.932696] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8447.464925] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8447.465368] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8447.470724] Lustre: Skipped 1 previous similar message [ 8447.476320] Lustre: Skipped 3 previous similar messages [ 8450.687579] Lustre: server umount lustre-MDT0000 complete [ 8453.075858] LustreError: 252523:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788810144 with bad export cookie 8585938443142586431 [ 8453.083237] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8453.090427] LustreError: 252523:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8453.294308] Lustre: server umount lustre-MDT0001 complete [ 8464.944666] Lustre: server umount lustre-OST0000 complete [ 8477.687815] Lustre: server umount lustre-OST0001 complete [ 8485.604980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8490.898707] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8495.968575] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 8501.526625] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 8511.990669] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 8517.041384] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8517.051249] Lustre: lustre-MDT0000: reset Object Index mappings [ 8517.253256] LustreError: 260035:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8517.259627] LustreError: 260035:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 8517.294957] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8519.386083] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8522.723920] LustreError: 260036:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8524.218720] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8524.234494] Lustre: lustre-MDT0001: reset Object Index mappings [ 8526.581030] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8528.051488] Lustre: 261176:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8531.733162] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8531.966291] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8531.972505] Lustre: Skipped 1 previous similar message [ 8533.995216] LustreError: 261533:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8534.997901] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8539.057451] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8539.110263] LustreError: 261755:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8542.028839] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8542.251667] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8542.255736] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 8546.487979] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8548.635924] Lustre: 263048:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8552.422268] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8552.427971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8552.430303] Lustre: Skipped 4 previous similar messages [ 8552.435543] Lustre: Skipped 3 previous similar messages [ 8557.133142] Lustre: server umount lustre-MDT0000 complete [ 8557.537976] LustreError: 260036:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8557.551476] LustreError: 260036:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 8559.123296] LustreError: 260017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788810250 with bad export cookie 8585938443142593221 [ 8559.125577] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8559.129856] LustreError: 260017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8559.299395] Lustre: server umount lustre-MDT0001 complete [ 8570.915841] Lustre: server umount lustre-OST0000 complete [ 8583.449405] Lustre: server umount lustre-OST0001 complete [ 8593.266838] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 15:44:44 (1788810284) [ 8602.157799] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [ 8607.276193] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 8626.595406] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8626.720343] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8626.736514] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8626.782553] Lustre: lustre-MDT0000: new disk, initializing [ 8626.822906] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8626.828338] Lustre: Skipped 1 previous similar message [ 8626.846384] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8628.715349] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8634.566573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8634.623984] Lustre: 267296:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8634.767635] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8637.006291] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8642.137816] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8644.874235] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8648.600801] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8648.757461] Lustre: lustre-OST0000: new disk, initializing [ 8648.760883] Lustre: Skipped 1 previous similar message [ 8648.763315] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8648.765671] Lustre: Skipped 2 previous similar messages [ 8648.767800] Lustre: 269260:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8650.625668] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8650.628946] Lustre: Skipped 1 previous similar message [ 8650.631972] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8650.693841] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8651.860525] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8657.513520] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8659.739195] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8659.849779] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8662.043958] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8662.174548] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8666.080646] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8666.083040] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8666.084242] LustreError: Skipped 1 previous similar message [ 8666.091602] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8666.094602] Lustre: Skipped 1 previous similar message [ 8669.280421] Lustre: server umount lustre-OST0000 complete [ 8675.810509] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8675.813305] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8675.819197] Lustre: Skipped 1 previous similar message [ 8675.821685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8677.470778] Lustre: server umount lustre-MDT0000 complete [ 8679.447920] LustreError: 267331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788810370 with bad export cookie 8585938443142595307 [ 8679.449925] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8679.455644] LustreError: 267331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8679.684140] Lustre: server umount lustre-MDT0001 complete [ 8689.021448] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 8693.823497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8694.017929] LustreError: 272188:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8694.038576] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8694.041408] Lustre: Skipped 2 previous similar messages [ 8696.004116] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8700.324177] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8702.976754] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8704.779602] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8708.279784] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8711.624904] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8716.178864] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8716.301389] Lustre: lustre-OST0001: new disk, initializing [ 8716.306770] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8716.311603] Lustre: 274494:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8718.236136] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8718.240693] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8718.300759] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8719.129395] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8724.654695] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8726.366481] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8728.444401] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 8728.455478] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 8734.690656] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8734.691764] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8734.699324] Lustre: Skipped 4 previous similar messages [ 8734.701714] Lustre: Skipped 4 previous similar messages [ 8734.840179] Lustre: server umount lustre-MDT0000 complete [ 8736.520636] LustreError: 272171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788810427 with bad export cookie 8585938443142596189 [ 8736.525900] LustreError: 272171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8736.526497] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8738.708929] Lustre: server umount lustre-MDT0001 complete [ 8749.901736] Lustre: server umount lustre-OST0000 complete [ 8760.955270] Lustre: server umount lustre-OST0001 complete [ 8770.009267] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 8775.307597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8775.504789] LustreError: 277831:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8775.517212] LustreError: 277831:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8775.557423] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8775.559673] Lustre: Skipped 3 previous similar messages [ 8777.864592] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8782.238558] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8784.824488] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8786.470719] Lustre: 278971:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8790.241151] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8793.586790] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8798.421796] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8800.635417] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 8800.650347] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 8801.581847] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8806.098380] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8807.837140] Lustre: 280848:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8810.656381] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 8810.669037] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 8817.509235] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 15:48:29 (1788810509) [ 8831.468989] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8831.470126] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8831.473545] Lustre: Skipped 3 previous similar messages [ 8831.476232] Lustre: Skipped 2 previous similar messages [ 8844.255128] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8844.390947] Lustre: server umount lustre-MDT0000 complete [ 8846.378069] LustreError: 277814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788810537 with bad export cookie 8585938443142602559 [ 8846.382259] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8846.384040] LustreError: 277814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8846.816656] LustreError: 279347:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8846.828169] LustreError: 279347:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 8852.666835] Lustre: server umount lustre-MDT0001 complete [ 8868.831291] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8868.879859] Lustre: server umount lustre-OST0000 complete [ 8885.215088] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8885.319683] Lustre: server umount lustre-OST0001 complete [ 8889.052884] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [ 8894.217457] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 8917.598293] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 8922.894542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8923.047542] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8923.071262] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8923.124219] Lustre: lustre-MDT0000: new disk, initializing [ 8923.171921] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8923.176231] Lustre: Skipped 3 previous similar messages [ 8923.185298] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8925.226843] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8931.115692] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8931.183256] Lustre: 285373:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8931.326299] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8933.578334] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8940.306964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8940.478172] Lustre: 287002:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8942.358628] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8943.714970] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8949.797502] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8949.887214] Lustre: 287873:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8951.283375] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8953.017330] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8958.794433] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8960.616187] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8960.619713] Lustre: Skipped 1 previous similar message [ 8967.225858] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 15:50:58 (1788810658) [ 8971.745155] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8971.749393] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8971.753866] Lustre: Skipped 5 previous similar messages [ 8971.756159] Lustre: Skipped 14 previous similar messages [ 8974.632180] Lustre: server umount lustre-MDT0000 complete [ 8976.689653] LustreError: 286168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788810668 with bad export cookie 8585938443142610504 [ 8976.692288] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8976.697977] LustreError: 286168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8976.864954] LustreError: 287181:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8976.873204] LustreError: 287181:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 8976.928614] Lustre: server umount lustre-MDT0001 complete [ 8989.215035] Lustre: server umount lustre-OST0000 complete [ 8991.291551] Lustre: server umount lustre-OST0001 complete [ 8996.120428] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8998.855363] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9003.257432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9005.673271] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9009.222450] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9011.556363] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9014.646798] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9017.294888] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9020.408336] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9037.839773] loop1: detected capacity change from 0 to 204800000 [ 9037.943919] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9037.948562] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9037.952780] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9040.348901] loop1: detected capacity change from 0 to 204800000 [ 9043.222023] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9043.272219] LustreError: 290104:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs7348: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [ 9043.281782] LustreError: 290104:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs7348 [ 9043.286234] LustreError: 290104:0:(mgs_handler.c:517:mgs_target_reg()) fs7348-MDT0001: Error checking target index: rc = -13 [ 9043.292119] LustreError: fs7348-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 9043.296520] LustreError: 293848:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -13 [ 9043.300166] LustreError: 293848:0:(tgt_mount.c:2128:server_put_super()) no obd fs7348-MDT0001 [ 9043.304169] LustreError: 293848:0:(tgt_mount.c:116:server_deregister_mount()) fs7348-MDT0001 not registered [ 9043.326118] Lustre: server umount fs7348-MDT0001 complete [ 9043.328483] LustreError: 293848:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 9047.677790] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9047.721620] Lustre: Setting parameter fs7348-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs7348-MDT0001 [ 9047.749144] Lustre: srv-fs7348-MDT0001: No data found on store. Initialize space. [ 9047.751217] Lustre: Skipped 4 previous similar messages [ 9047.795647] Lustre: fs7348-MDT0001: new disk, initializing [ 9047.797752] Lustre: Skipped 3 previous similar messages [ 9047.866103] Lustre: srv-fs7348-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 9047.869375] Lustre: 294313:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs7348-MDT0001-mdtlov: can't create new llog: rc = -115 [ 9047.872178] LustreError: 294313:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs7348-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 9050.027886] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9053.544266] Lustre: server umount fs7348-MDT0001 complete [ 9059.315183] loop1: detected capacity change from 0 to 204800000 [ 9061.971173] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9062.135072] Lustre: 295880:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs7348-MDT0001-mdtlov: can't create new llog: rc = -115 [ 9062.138209] LustreError: 295880:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs7348-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 9063.999208] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9066.466216] Lustre: server umount fs7348-MDT0001 complete [ 9072.211870] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.4@tcp (stopping) [ 9072.215032] Lustre: Skipped 1 previous similar message [ 9074.147399] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9074.152742] Lustre: Skipped 2 previous similar messages [ 9077.383476] Lustre: server umount lustre-MDT0000 complete [ 9081.457469] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9081.528405] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9083.933417] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9086.951568] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9087.527681] loop1: detected capacity change from 0 to 204800000 [ 9090.486451] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9090.695862] Lustre: 298323:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs7348-MDT0001-mdtlov: can't create new llog: rc = -115 [ 9090.700488] LustreError: 298323:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs7348-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 9092.771343] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9095.184516] Lustre: server umount fs7348-MDT0001 complete [ 9107.429984] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9107.431496] LustreError: 296095:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9107.442169] LustreError: 296095:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 54 previous similar messages [ 9111.607924] Lustre: server umount lustre-OST0000 complete [ 9117.664188] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9119.679991] Lustre: server umount lustre-MDT0000 complete [ 9121.684727] LustreError: 290096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788810812 with bad export cookie 8585938443142613185 [ 9121.685796] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9121.689592] LustreError: 290096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9121.878112] Lustre: server umount lustre-MDT0001 complete [ 9127.130498] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 9128.645678] Key type lgssc unregistered [ 9128.861597] LNet: 300661:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9128.864946] LNetError: 300661:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9128.875619] LNet: Removed LNI 192.168.201.104@tcp [ 9129.326139] Key type .llcrypt unregistered [ 9129.328748] Key type ._llcrypt unregistered [ 9140.207632] Key type ._llcrypt registered [ 9140.211867] Key type .llcrypt registered [ 9140.255275] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 9140.790304] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9140.848515] alg: No test for adler32 (adler32-zlib) [ 9141.730965] Lustre: Lustre: Build Version: 2.17.58_39_gb44c962 [ 9141.861122] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 9143.487372] Key type lgssc registered [ 9144.082810] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9147.383490] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9147.393159] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9148.593753] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9150.682660] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9154.679072] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9154.864382] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9156.727469] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9160.009668] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9162.751571] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9166.368341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9166.624982] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9169.668589] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9172.772899] LustreError: 304358:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9172.783945] LustreError: 304358:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9173.385565] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9175.650777] LustreError: 303733:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9178.084051] LustreError: 304358:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9180.139394] LustreError: 304358:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9180.146476] LustreError: 304358:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9185.251418] LustreError: 303733:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9185.257848] LustreError: 303733:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9186.637359] loop1: detected capacity change from 0 to 204800000 [ 9186.748724] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9186.752697] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9186.756742] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9188.991667] loop1: detected capacity change from 0 to 204800000 [ 9191.558535] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9191.625814] Lustre: fs25952-OST0000: new disk, initializing [ 9191.630547] Lustre: srv-fs25952-OST0000: No data found on store. Initialize space. [ 9191.633906] Lustre: 305639:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9191.677401] Lustre: fs25952-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9194.319101] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9195.502993] LustreError: 304409:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9195.509956] LustreError: 304409:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 9199.089972] Lustre: server umount fs25952-OST0000 complete [ 9204.474774] loop1: detected capacity change from 0 to 204800000 [ 9206.801497] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9206.897437] Lustre: fs25952-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9209.472811] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9213.185330] loop2: detected capacity change from 0 to 204800000 [ 9213.289310] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9213.295336] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9213.304589] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9215.703236] loop2: detected capacity change from 0 to 204800000 [ 9215.970659] LustreError: 303736:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9215.980337] LustreError: 303736:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [ 9218.016379] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9218.052038] LustreError: fs25952-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [ 9218.057177] LustreError: fs25952-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 9218.060835] LustreError: 308677:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -13 [ 9218.063785] LustreError: 308677:0:(tgt_mount.c:2128:server_put_super()) no obd fs25952-OST0001 [ 9218.066274] LustreError: 308677:0:(tgt_mount.c:116:server_deregister_mount()) fs25952-OST0001 not registered [ 9218.081931] Lustre: server umount fs25952-OST0001 complete [ 9218.084120] LustreError: 308677:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 9222.768647] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9222.836414] Lustre: fs25952-OST0001: new disk, initializing [ 9222.840756] Lustre: srv-fs25952-OST0001: No data found on store. Initialize space. [ 9222.845263] Lustre: 309228:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9222.891527] Lustre: fs25952-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9225.571472] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9230.139439] Lustre: server umount fs25952-OST0000 complete [ 9234.978615] Lustre: server umount fs25952-OST0001 complete [ 9240.892546] loop1: detected capacity change from 0 to 204800000 [ 9243.786743] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9243.920644] Lustre: fs25952-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9247.171651] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9250.794868] loop2: detected capacity change from 0 to 204800000 [ 9251.819454] LustreError: 303735:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9251.827363] LustreError: 303735:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 20 previous similar messages [ 9253.619358] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9256.916602] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9256.940884] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [ 9261.293638] Lustre: server umount fs25952-OST0000 complete [ 9265.982317] Lustre: server umount fs25952-OST0001 complete [ 9267.176914] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [ 9272.295432] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9272.296535] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9272.303287] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9272.309904] Lustre: Skipped 1 previous similar message [ 9277.411554] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9277.420371] Lustre: Skipped 1 previous similar message [ 9282.536751] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9282.547251] Lustre: Skipped 1 previous similar message [ 9286.111126] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9286.219661] Lustre: server umount lustre-OST0000 complete [ 9292.769539] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9292.771405] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9292.776904] Lustre: Skipped 2 previous similar messages [ 9294.567644] Lustre: server umount lustre-MDT0000 complete [ 9296.428552] LustreError: 301888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788810987 with bad export cookie 18326502050056055345 [ 9296.435391] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9296.437822] LustreError: 301888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9296.669470] Lustre: server umount lustre-MDT0001 complete [ 9301.912892] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 9303.312577] Key type lgssc unregistered [ 9303.482528] LNet: 315140:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9303.489239] LNetError: 315140:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9303.501368] LNet: Removed LNI 192.168.201.104@tcp [ 9303.987106] Key type .llcrypt unregistered [ 9303.988626] Key type ._llcrypt unregistered [ 9320.274277] Key type ._llcrypt registered [ 9320.275784] Key type .llcrypt registered [ 9320.318392] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [ 9328.651458] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 9329.194402] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9329.203781] alg: No test for adler32 (adler32-zlib) [ 9330.094665] Lustre: Lustre: Build Version: 2.17.58_39_gb44c962 [ 9330.192467] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 9331.791211] Key type lgssc registered [ 9332.325331] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9347.962809] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9347.970291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9349.086543] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9349.109584] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9349.163952] Lustre: lustre-MDT0000: new disk, initializing [ 9349.208375] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9349.224497] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9350.883635] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9355.960883] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9356.007332] Lustre: 318967:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9356.030853] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9356.033322] Lustre: Skipped 1 previous similar message [ 9356.073090] Lustre: lustre-MDT0001: new disk, initializing [ 9356.113815] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9356.132116] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9356.138369] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9358.349356] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9363.301902] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9365.795917] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9368.686125] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9368.870823] Lustre: lustre-OST0000: new disk, initializing [ 9368.874513] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9368.879945] Lustre: 320937:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9368.961178] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9371.908838] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9374.285302] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9374.292953] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9374.413083] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9380.051631] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9381.929543] Lustre: Mounted lustre-client - version 2.17.58_39_gb44c962 [ 9382.882327] Lustre: Unmounted lustre-client [ 9392.103845] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9392.121582] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9397.222235] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9397.229752] Lustre: Skipped 2 previous similar messages [ 9397.703830] Lustre: server umount lustre-OST0000 complete [ 9402.336745] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9402.340288] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9402.348180] Lustre: Skipped 1 previous similar message [ 9402.349882] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9407.460127] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9407.475638] Lustre: Skipped 2 previous similar messages [ 9408.164840] Lustre: server umount lustre-MDT0000 complete [ 9412.063805] LustreError: 319763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788811103 with bad export cookie 6199512144480793226 [ 9412.064737] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9412.075828] LustreError: 319763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9412.493694] Lustre: server umount lustre-MDT0001 complete [ 9420.947713] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 9423.211813] Key type lgssc unregistered [ 9435.743551] LNet: 323095:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9435.751678] LNetError: 323095:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9512.553710] LNet: Removed LNI 192.168.201.104@tcp [ 9513.045372] Key type .llcrypt unregistered [ 9513.047728] Key type ._llcrypt unregistered [ 9524.514949] Key type ._llcrypt registered [ 9524.516806] Key type .llcrypt registered [ 9524.584620] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [ 9534.672390] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 9535.375104] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9535.398035] alg: No test for adler32 (adler32-zlib) [ 9536.354033] Lustre: Lustre: Build Version: 2.17.58_39_gb44c962 [ 9536.505027] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 9538.137549] Key type lgssc registered [ 9538.852041] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9557.173508] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 16:00:48 (1788811248) [ 9561.186838] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9561.201792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9562.357293] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9562.378613] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9562.423560] Lustre: lustre-MDT0000: new disk, initializing [ 9562.465709] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9562.477371] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9564.973220] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9571.827044] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9571.899615] Lustre: 326291:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9571.937899] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9571.940757] Lustre: Skipped 1 previous similar message [ 9571.986612] Lustre: lustre-MDT0001: new disk, initializing [ 9572.067606] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9572.091173] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9572.099563] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9574.229993] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9579.978508] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9583.249231] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9587.510750] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9587.692741] Lustre: lustre-OST0000: new disk, initializing [ 9587.694887] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9587.699731] Lustre: 328266:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9587.738313] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9590.907937] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9596.491787] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9597.477072] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9597.489626] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9597.528913] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9607.650847] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9607.656549] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9607.659641] Lustre: Skipped 1 previous similar message [ 9607.660472] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9607.663487] Lustre: Skipped 1 previous similar message [ 9612.052281] Lustre: server umount lustre-OST0000 complete [ 9617.896559] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9617.899312] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9617.905859] Lustre: Skipped 1 previous similar message [ 9623.009775] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9623.013595] Lustre: Skipped 2 previous similar messages [ 9628.127165] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9628.128709] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9628.144439] Lustre: Skipped 1 previous similar message [ 9628.259987] Lustre: server umount lustre-MDT0000 complete [ 9630.650132] LustreError: 326281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788811321 with bad export cookie 11410009397027159879 [ 9630.652737] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9630.657124] LustreError: 326281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9630.919197] Lustre: server umount lustre-MDT0001 complete [ 9637.486939] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 9639.229983] Key type lgssc unregistered [ 9639.448599] LNet: 330171:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9639.454104] LNetError: 330171:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9640.485317] LNet: Removed LNI 192.168.201.104@tcp [ 9641.094944] Key type .llcrypt unregistered [ 9641.096981] Key type ._llcrypt unregistered [ 9654.388053] Key type ._llcrypt registered [ 9654.389464] Key type .llcrypt registered [ 9654.476528] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 9655.161052] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9655.273501] alg: No test for adler32 (adler32-zlib) [ 9656.232989] Lustre: Lustre: Build Version: 2.17.58_39_gb44c962 [ 9656.379350] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 9658.031165] Key type lgssc registered [ 9658.693634] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9663.652967] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9663.666717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9664.906529] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9667.507712] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9672.677178] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9672.896839] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9675.587730] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9680.014387] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9683.514864] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9687.865174] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9688.051378] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9692.207830] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9696.530914] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9703.769306] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9704.748981] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9708.521043] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9708.527928] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9708.532112] Lustre: Skipped 2 previous similar messages [ 9708.537708] Lustre: Skipped 1 previous similar message [ 9709.147758] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.4@tcp (stopping) [ 9709.152241] Lustre: Skipped 1 previous similar message [ 9713.325030] Lustre: server umount lustre-MDT0000 complete [ 9713.633629] LustreError: 332157:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9713.660049] LustreError: 332157:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 9714.247889] LustreError: 331420:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9715.644327] LustreError: 331407:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788811406 with bad export cookie 5591458423918213169 [ 9715.644885] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9715.654404] LustreError: 331407:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9718.756592] LustreError: 331422:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9718.756881] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9718.770674] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9720.802077] LustreError: 331420:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9720.802839] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9720.810160] LustreError: 331420:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9720.821678] Lustre: Skipped 1 previous similar message [ 9721.965227] Lustre: server umount lustre-MDT0001 complete [ 9730.354978] Lustre: server umount lustre-OST0000 complete [ 9735.340531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9738.228276] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9740.259256] Lustre: 335602:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.104@tcp [ 9741.763775] Lustre: server umount MGS complete [ 9747.052144] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9747.298306] LustreError: 336173:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9747.337932] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9749.970963] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9754.422070] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9754.616413] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9756.797628] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9761.401130] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9765.110861] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9768.970430] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9769.796304] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [ 9772.895657] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9775.846949] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 9780.195860] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9780.207739] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9780.219659] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9782.244561] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9782.252914] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9784.139728] Lustre: server umount lustre-OST0000 complete [ 9787.361462] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9787.370530] Lustre: Skipped 1 previous similar message [ 9792.413975] Lustre: server umount lustre-MDT0000 complete [ 9792.483447] LustreError: 336169:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9792.491047] LustreError: 336169:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9794.075342] LustreError: 336155:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788811485 with bad export cookie 5591458423918214002 [ 9794.078740] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9794.080244] LustreError: 336155:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9794.227331] Lustre: server umount lustre-MDT0001 complete [ 9798.567567] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 9799.803271] Key type lgssc unregistered [ 9799.978988] LNet: 339602:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9799.985432] LNetError: 339602:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9799.993459] LNet: Removed LNI 192.168.201.104@tcp [ 9800.378369] Key type .llcrypt unregistered [ 9800.380117] Key type ._llcrypt unregistered [ 9808.125937] Key type ._llcrypt registered [ 9808.127138] Key type .llcrypt registered [ 9808.181934] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [ 9817.174129] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 9817.701291] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9817.733115] alg: No test for adler32 (adler32-zlib) [ 9818.647729] Lustre: Lustre: Build Version: 2.17.58_39_gb44c962 [ 9818.799511] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 9820.408494] Key type lgssc registered [ 9821.003294] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9840.898505] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9840.910120] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9842.103572] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9842.127736] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9842.181380] Lustre: lustre-MDT0000: new disk, initializing [ 9842.232056] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9842.244311] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9844.313737] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9850.333980] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9850.423771] Lustre: 342734:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9850.457545] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9850.460567] Lustre: Skipped 1 previous similar message [ 9850.522807] Lustre: lustre-MDT0001: new disk, initializing [ 9850.578901] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9850.605664] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9850.614623] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9852.797031] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9858.570929] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9861.661184] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9865.453112] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9865.604189] Lustre: lustre-OST0000: new disk, initializing [ 9865.608300] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9865.616260] Lustre: 344701:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9865.690157] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9868.850163] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9871.880781] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9871.884983] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9871.929855] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9874.090389] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9876.367231] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9876.484276] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9878.655595] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9878.775338] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9882.084823] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9882.085378] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9882.090329] LustreError: Skipped 1 previous similar message [ 9882.096380] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9882.101142] Lustre: Skipped 1 previous similar message [ 9887.201376] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9887.205902] Lustre: Skipped 1 previous similar message [ 9892.323789] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9892.326873] Lustre: Skipped 1 previous similar message [ 9894.367176] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9894.491298] Lustre: server umount lustre-OST0000 complete [ 9896.928922] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9896.934723] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9896.942753] Lustre: Skipped 1 previous similar message [ 9896.947322] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9902.561591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9902.565490] Lustre: Skipped 3 previous similar messages [ 9911.263406] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9911.346929] Lustre: server umount lustre-MDT0000 complete [ 9913.505561] LustreError: 342726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788811604 with bad export cookie 14901672613087636779 [ 9913.506413] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9913.511506] LustreError: 342726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9913.727995] Lustre: server umount lustre-MDT0001 complete [ 9917.954328] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 16:06:49 (1788811609) [ 9925.655210] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [ 9930.505098] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 9949.943026] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9950.078784] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9950.091885] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9950.139479] Lustre: lustre-MDT0000: new disk, initializing [ 9950.187595] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9950.210512] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9952.645585] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9958.429139] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9958.507476] Lustre: 349457:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9958.541830] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9958.545060] Lustre: Skipped 1 previous similar message [ 9958.596217] Lustre: lustre-MDT0001: new disk, initializing [ 9958.647524] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9958.674383] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9958.680608] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9960.956736] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9966.138863] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9969.089947] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9972.544964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9972.697788] Lustre: lustre-OST0000: new disk, initializing [ 9972.700358] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9972.703201] Lustre: 351422:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9972.753462] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9976.052791] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9979.454928] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9979.460345] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9979.550754] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9981.776232] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10000.354461] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10000.356567] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10000.357663] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10000.357669] Lustre: Skipped 2 previous similar messages [10000.364199] Lustre: Skipped 2 previous similar messages [10013.151154] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10013.264591] Lustre: server umount lustre-OST0000 complete [10015.199952] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10015.204093] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10020.833051] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10020.846564] Lustre: Skipped 8 previous similar messages [10021.513302] Lustre: server umount lustre-MDT0000 complete [10023.749509] LustreError: 349449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788811715 with bad export cookie 14901672613087637661 [10023.752155] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10023.756277] LustreError: 349449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10023.941576] Lustre: server umount lustre-MDT0001 complete [10031.421768] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [10033.175324] Key type lgssc unregistered [10033.384544] LNet: 353573:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10033.389275] LNetError: 353573:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10033.411157] LNet: Removed LNI 192.168.201.104@tcp [10034.035416] Key type .llcrypt unregistered [10034.040862] Key type ._llcrypt unregistered [10082.105525] Key type ._llcrypt registered [10082.107135] Key type .llcrypt registered [10082.162286] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [10082.325073] Key type .llcrypt unregistered [10082.328589] Key type ._llcrypt unregistered [10119.208327] Key type ._llcrypt registered [10119.209556] Key type .llcrypt registered [10119.266696] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [10119.402559] Key type .llcrypt unregistered [10119.404596] Key type ._llcrypt unregistered [10132.070418] Key type ._llcrypt registered [10132.071877] Key type .llcrypt registered [10132.119250] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [10132.239294] Key type .llcrypt unregistered [10132.240743] Key type ._llcrypt unregistered [10145.831144] Key type ._llcrypt registered [10145.832796] Key type .llcrypt registered [10145.875591] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [10146.004486] Key type .llcrypt unregistered [10146.006058] Key type ._llcrypt unregistered [10179.902522] Key type ._llcrypt registered [10179.903904] Key type .llcrypt registered [10179.957148] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [10180.086373] Key type .llcrypt unregistered [10180.088398] Key type ._llcrypt unregistered [10193.932130] Key type ._llcrypt registered [10193.933623] Key type .llcrypt registered [10193.980888] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [10194.110411] Key type .llcrypt unregistered [10194.112082] Key type ._llcrypt unregistered [10207.023806] Key type ._llcrypt registered [10207.026640] Key type .llcrypt registered [10207.085719] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [10207.223790] Key type .llcrypt unregistered [10207.225724] Key type ._llcrypt unregistered [10221.251877] Key type ._llcrypt registered [10221.253451] Key type .llcrypt registered [10221.303272] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [10221.439216] Key type .llcrypt unregistered [10221.440608] Key type ._llcrypt unregistered [10234.913611] Key type ._llcrypt registered [10234.915263] Key type .llcrypt registered [10234.959065] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [10235.093353] Key type .llcrypt unregistered [10235.094674] Key type ._llcrypt unregistered [10248.143454] Key type ._llcrypt registered [10248.144605] Key type .llcrypt registered [10248.189724] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [10248.327291] Key type .llcrypt unregistered [10248.328758] Key type ._llcrypt unregistered [10274.446195] Key type ._llcrypt registered [10274.447139] Key type .llcrypt registered [10274.488521] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [10274.601462] Key type .llcrypt unregistered [10274.602647] Key type ._llcrypt unregistered [10286.887941] Key type ._llcrypt registered [10286.889420] Key type .llcrypt registered [10286.932710] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [10287.049695] Key type .llcrypt unregistered [10287.051647] Key type ._llcrypt unregistered [10300.931749] Key type ._llcrypt registered [10300.932957] Key type .llcrypt registered [10300.975894] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [10301.106453] Key type .llcrypt unregistered [10301.107529] Key type ._llcrypt unregistered [10323.927837] Key type ._llcrypt registered [10323.929168] Key type .llcrypt registered [10323.975752] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [10331.424339] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [10331.777228] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10331.799049] alg: No test for adler32 (adler32-zlib) [10332.695352] Lustre: Lustre: Build Version: 2.17.58_39_gb44c962 [10332.800489] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [10334.383183] Key type lgssc registered [10334.859061] Lustre: Echo OBD driver; http://www.lustre.org/ [10349.255566] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10349.281689] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10350.394841] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [10350.414694] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [10350.462950] Lustre: tst250fs-MDT0000: new disk, initializing [10350.494091] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10350.502417] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10352.274163] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [10357.629639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10357.689236] Lustre: 373442:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10357.717608] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [10357.722184] Lustre: Skipped 1 previous similar message [10357.765822] Lustre: tst250fs-MDT0001: new disk, initializing [10357.845086] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10357.864645] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10357.870282] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10359.882248] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [10362.736367] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10365.512120] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10365.623905] Lustre: tst250fs-OST0000: new disk, initializing [10365.625893] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [10365.628994] Lustre: 375064:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10365.667340] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10367.164490] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10367.168542] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10367.225603] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10368.046698] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [10372.968850] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10373.051559] Lustre: tst250fs-OST0001: new disk, initializing [10373.054255] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [10373.057808] Lustre: 375944:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10373.115942] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10374.471934] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10374.475803] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10374.514884] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10375.421282] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [10378.310973] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [10393.571949] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10393.578488] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10393.594406] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [10395.104505] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [10395.110411] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10400.227407] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [10400.230033] Lustre: Skipped 6 previous similar messages [10404.319436] Lustre: tst250fs-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10404.458147] Lustre: server umount tst250fs-MDT0000 complete [10407.451677] LustreError: 375942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788812098 with bad export cookie 2397333906438235480 [10407.461579] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10407.469796] LustreError: 375942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10407.667669] Lustre: server umount tst250fs-MDT0001 complete [10421.427781] Lustre: server umount tst250fs-OST0000 complete [10435.094091] Lustre: server umount tst250fs-OST0001 complete [10454.266412] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [10460.102599] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [10479.094725] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 16:16:10 (1788812170) [10480.040834] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [10490.198712] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [10496.535926] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [10516.903354] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 10246 sec ========= 16:16:48 (1788812208) [10518.243930] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 16:16:49 (1788812209) === [10520.803265] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 16:16:52 (1788812212) === [10546.556932] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [10548.077727] Key type lgssc unregistered [10548.240553] LNet: 385059:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10548.243837] LNetError: 385059:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10549.286716] LNet: Removed LNI 192.168.201.104@tcp [10549.894119] Key type .llcrypt unregistered [10549.895087] Key type ._llcrypt unregistered