[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 506724619 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003401] x2apic enabled [ 0.004011] Switched APIC routing to physical x2apic. [ 0.005017] kvm-guest: setup PV IPIs [ 0.008427] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010012] pid_max: default: 32768 minimum: 301 [ 0.011105] LSM: Security Framework initializing [ 0.012042] Yama: becoming mindful. [ 0.013037] SELinux: Initializing. [ 0.014086] *** VALIDATE selinux *** [ 0.022201] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027325] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029034] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031127] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032095] *** VALIDATE tmpfs *** [ 0.033360] *** VALIDATE proc *** [ 0.034190] *** VALIDATE cgroup *** [ 0.035006] *** VALIDATE cgroup2 *** [ 0.036193] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037114] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039034] Spectre V2 : User space: Vulnerable [ 0.040009] Speculative Store Bypass: Vulnerable [ 0.042979] debug: unmapping init [mem 0xffffffff85059000-0xffffffff85060fff] [ 0.045992] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046711] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047027] ... version: 2 [ 0.048010] ... bit width: 48 [ 0.049012] ... generic registers: 4 [ 0.050010] ... value mask: 0000ffffffffffff [ 0.051013] ... max period: 00007fffffffffff [ 0.052013] ... fixed-purpose events: 3 [ 0.052832] ... event mask: 000000070000000f [ 0.053252] rcu: Hierarchical SRCU implementation. [ 0.055270] smp: Bringing up secondary CPUs ... [ 0.056586] x86: Booting SMP configuration: [ 0.057036] .... node #0, CPUs: #1 #2 #3 [ 0.065133] smp: Brought up 1 node, 4 CPUs [ 0.067039] smpboot: Max logical packages: 1 [ 0.067887] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.147699] node 0 deferred pages initialised in 78ms [ 0.149097] devtmpfs: initialized [ 0.150236] x86/mm: Memory block size: 128MB [ 0.154611] gcov: version magic: 0x41383552 [ 0.156334] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.159083] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.160285] pinctrl core: initialized pinctrl subsystem [ 0.161318] [ 0.161756] ************************************************************* [ 0.162013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.163014] ** ** [ 0.164013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.165013] ** ** [ 0.166013] ** This means that this kernel is built to expose internal ** [ 0.167013] ** IOMMU data structures, which may compromise security on ** [ 0.168015] ** your system. ** [ 0.169017] ** ** [ 0.170014] ** If you see this message and you are not debugging the ** [ 0.171014] ** kernel, report this immediately to your vendor! ** [ 0.172008] ** ** [ 0.173015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.174009] ************************************************************* [ 0.175725] NET: Registered protocol family 16 [ 0.176476] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.177063] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.178068] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.179445] cpuidle: using governor menu [ 0.181618] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.184457] PCI: Using configuration type 1 for base access [ 0.187134] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.196121] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.197048] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.199055] cryptd: max_cpu_qlen set to 1000 [ 0.201096] ACPI: Added _OSI(Module Device) [ 0.202011] ACPI: Added _OSI(Processor Device) [ 0.203012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.204012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.208547] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.211648] ACPI: Interpreter enabled [ 0.212074] ACPI: PM: (supports S0 S3 S4 S5) [ 0.213013] ACPI: Using IOAPIC for interrupt routing [ 0.214094] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.215410] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.225676] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.226053] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.227026] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.228091] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.230549] acpiphp: Slot [2] registered [ 0.231117] acpiphp: Slot [5] registered [ 0.232103] acpiphp: Slot [6] registered [ 0.233135] acpiphp: Slot [7] registered [ 0.234108] acpiphp: Slot [8] registered [ 0.235095] acpiphp: Slot [9] registered [ 0.236103] acpiphp: Slot [10] registered [ 0.237113] acpiphp: Slot [3] registered [ 0.238105] acpiphp: Slot [4] registered [ 0.239095] acpiphp: Slot [11] registered [ 0.240093] acpiphp: Slot [12] registered [ 0.241101] acpiphp: Slot [13] registered [ 0.242080] acpiphp: Slot [14] registered [ 0.243095] acpiphp: Slot [15] registered [ 0.244113] acpiphp: Slot [16] registered [ 0.245099] acpiphp: Slot [17] registered [ 0.246114] acpiphp: Slot [18] registered [ 0.247106] acpiphp: Slot [19] registered [ 0.248090] acpiphp: Slot [20] registered [ 0.249093] acpiphp: Slot [21] registered [ 0.250081] acpiphp: Slot [22] registered [ 0.251084] acpiphp: Slot [23] registered [ 0.252103] acpiphp: Slot [24] registered [ 0.253096] acpiphp: Slot [25] registered [ 0.254088] acpiphp: Slot [26] registered [ 0.255081] acpiphp: Slot [27] registered [ 0.256085] acpiphp: Slot [28] registered [ 0.257107] acpiphp: Slot [29] registered [ 0.258106] acpiphp: Slot [30] registered [ 0.260085] acpiphp: Slot [31] registered [ 0.261065] PCI host bridge to bus 0000:00 [ 0.262022] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.263022] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.264018] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.265023] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.266023] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.267047] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.268173] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.270093] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.272063] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.276811] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.278427] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.279022] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.280013] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.281015] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.282642] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.285221] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.288049] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.292481] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.298016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.315020] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.320013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.328618] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.335016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.342016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.369019] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.390269] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.397018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.403017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.418017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.427804] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.441018] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.458025] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.503019] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.515867] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.529018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.544026] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.573020] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.593995] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.607034] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.618019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.641025] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.651272] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.659019] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.673018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.693020] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.702206] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.704227] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.706472] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.708261] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.709131] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.714108] iommu: Default domain type: Passthrough [ 0.715579] SCSI subsystem initialized [ 0.716000] ACPI: bus type USB registered [ 0.717081] usbcore: registered new interface driver usbfs [ 0.719086] usbcore: registered new interface driver hub [ 0.721089] usbcore: registered new device driver usb [ 0.723163] pps_core: LinuxPPS API ver. 1 registered [ 0.724063] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.727053] PTP clock support registered [ 0.729142] EDAC MC: Ver: 3.0.0 [ 0.731130] PCI: Using ACPI for IRQ routing [ 0.732567] NetLabel: Initializing [ 0.734013] NetLabel: domain hash size = 128 [ 0.734844] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.736097] NetLabel: unlabeled traffic allowed by default [ 0.738123] vgaarb: loaded [ 0.740275] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.742016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.747139] clocksource: Switched to clocksource kvm-clock [ 0.864568] VFS: Disk quotas dquot_6.6.0 [ 0.866171] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.868583] *** VALIDATE ramfs *** [ 0.869807] *** VALIDATE hugetlbfs *** [ 0.871234] pnp: PnP ACPI init [ 0.873538] pnp: PnP ACPI: found 6 devices [ 0.906136] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.909369] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.911722] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.914332] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.917156] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.919769] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.923325] NET: Registered protocol family 2 [ 0.926438] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.931165] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.934339] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.939926] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.943566] TCP: Hash tables configured (established 65536 bind 65536) [ 0.946443] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.950092] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.953023] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.956197] NET: Registered protocol family 1 [ 0.959284] RPC: Registered named UNIX socket transport module. [ 0.960825] RPC: Registered udp transport module. [ 0.962262] RPC: Registered tcp transport module. [ 0.963360] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.964695] NET: Registered protocol family 44 [ 0.966270] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.968482] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.970632] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.971948] PCI: CLS 0 bytes, default 64 [ 0.972930] Unpacking initramfs... [ 2.342111] debug: unmapping init [mem 0xffff9e833cc54000-0xffff9e833ffbffff] [ 2.345258] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.346802] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.348945] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.804818] Initialise system trusted keyrings [ 2.806764] Key type blacklist registered [ 2.808242] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.818203] zbud: loaded [ 2.821239] *** VALIDATE nfs *** [ 2.821969] *** VALIDATE nfs4 *** [ 2.823404] pstore: using deflate compression [ 2.826443] Platform Keyring initialized [ 2.924628] NET: Registered protocol family 38 [ 2.926448] Key type asymmetric registered [ 2.928207] Asymmetric key parser 'x509' registered [ 2.930129] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.933219] io scheduler mq-deadline registered [ 2.934899] io scheduler kyber registered [ 2.936727] io scheduler bfq registered [ 2.938688] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.943749] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.946326] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.948832] ACPI: Power Button [PWRF] [ 2.953282] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.959150] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.972348] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.982502] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.001544] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.026968] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.055773] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.059088] Non-volatile memory driver v1.3 [ 3.059919] Linux agpgart interface v0.103 [ 3.082376] virtio_blk virtio1: [vda] 149760 512-byte logical blocks (76.7 MB/73.1 MiB) [ 3.084839] vda: detected capacity change from 0 to 76677120 [ 3.098148] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.100289] vdb: detected capacity change from 0 to 1073741824 [ 3.112197] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.113865] vdc: detected capacity change from 0 to 2621440000 [ 3.126073] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.127733] vdd: detected capacity change from 0 to 2621440000 [ 3.138621] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.140557] vde: detected capacity change from 0 to 4294967296 [ 3.153314] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.155062] vdf: detected capacity change from 0 to 4294967296 [ 3.160872] libphy: Fixed MDIO Bus: probed [ 3.181373] usbcore: registered new interface driver usbserial_generic [ 3.183336] usbserial: USB Serial support registered for generic [ 3.185827] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.189752] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.191609] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.193778] mousedev: PS/2 mouse device common for all mice [ 3.197839] rtc_cmos 00:05: RTC can wake from S4 [ 3.201984] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.202751] rtc_cmos 00:05: registered as rtc0 [ 3.206681] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.209581] intel_pstate: CPU model not supported [ 3.212233] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.216410] hid: raw HID events driver (C) Jiri Kosina [ 3.218464] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.218523] usbcore: registered new interface driver usbhid [ 3.222405] usbhid: USB HID core driver [ 3.223948] drop_monitor: Initializing network drop monitor service [ 3.226418] Initializing XFRM netlink socket [ 3.228437] NET: Registered protocol family 10 [ 3.231502] Segment Routing with IPv6 [ 3.232301] NET: Registered protocol family 17 [ 3.234489] mpls_gso: MPLS GSO support [ 3.241589] RAS: Correctable Errors collector initialized. [ 3.243759] AVX version of gcm_enc/dec engaged. [ 3.244932] AES CTR mode by8 optimization enabled [ 3.313169] sched_clock: Marking stable (3313144410, 0)->(4259851303, -946706893) [ 3.315715] registered taskstats version 1 [ 3.317383] Loading compiled-in X.509 certificates [ 3.318600] zswap: loaded using pool lzo/zbud [ 3.341376] Key type big_key registered [ 3.351277] Key type encrypted registered [ 3.352751] ima: No TPM chip found, activating TPM-bypass! [ 3.354809] ima: Allocated hash algorithm: sha1 [ 3.356244] ima: No architecture policies found [ 3.359076] evm: Initialising EVM extended attributes: [ 3.360595] evm: security.selinux [ 3.361486] evm: security.ima [ 3.362282] evm: security.capability [ 3.363272] evm: HMAC attrs: 0x1 [ 3.365473] rtc_cmos 00:05: setting system clock to 2026-09-03 12:21:18 UTC (1788438078) [ 3.371183] debug: unmapping init [mem 0xffffffff86003000-0xffffffff861fffff] [ 3.373621] debug: unmapping init [mem 0xffffffff84d82000-0xffffffff85058fff] [ 3.382155] Write protecting the kernel read-only data: 28672k [ 3.385352] debug: unmapping init [mem 0xffffffff83403000-0xffffffff835fffff] [ 3.387646] debug: unmapping init [mem 0xffffffff83d14000-0xffffffff83dfffff] [ 3.422050] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.428769] systemd[1]: Detected virtualization kvm. [ 3.430289] systemd[1]: Detected architecture x86-64. [ 3.431739] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.459665] systemd[1]: No hostname configured. [ 3.461041] systemd[1]: Set hostname to . [ 3.462721] random: systemd: uninitialized urandom read (16 bytes read) [ 3.464706] systemd[1]: Initializing machine ID from random generator. [ 3.511492] random: ln: uninitialized urandom read (6 bytes read) [ 3.603988] random: systemd: uninitialized urandom read (16 bytes read) [ 3.605780] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.610068] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.613724] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Swap. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.171790] device-mapper: uevent: version 1.0.3 [ 4.173447] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.831353] random: fast init done [ 4.906410] scsi host0: ata_piix [ 4.918410] scsi host1: ata_piix [ 4.920794] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.925090] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 4.948226] virtio_net virtio0 ens2: renamed from eth0 [ 8.744166] dracut-initqueue[582]: RTNETLINK answers: File exists [ 9.868377] random: crng init done [ 9.869768] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.318506] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.460827] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.914240] SELinux: Disabled at runtime. [ 11.977134] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.985402] systemd[1]: Detected virtualization kvm. [ 11.987217] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 14.411737] systemd[1]: initrd-switch-root.service: Succeeded. [ 14.430747] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 14.442979] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 14.456680] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 14.464620] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 14.496164] systemd[1]: Starting Journal Service... Starting Journal Service... [ 14.512291] systemd[1]: proc-sys-fs-binfmt_misc.automount: Refusing to start, unit to trigger not loaded. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on RPCbind Server Activation Socket. Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-getty.slice. Mounting POSIX Message Queue File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Apply Kernel Variables... Starting udev Coldplug all Devices... [ OK ] Listening on Process Core Dump Socket. Activating swap /dev/disk/by-label/SWAP... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting Kernel Debug File System... [ OK ] Stopped target Switch Root. [ 15.095784] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target rpc_pipefs.target. Mounting Huge Pages File System... [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 16.307583] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 17.583632] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 17.646383] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 18.148348] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 18.255985] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit)[ 23.967308] Key type dns_resolver registered [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit)[ 24.938507] NFS: Registering the id_resolver key type [ 24.941904] Key type id_resolver registered [ 24.950588] Key type id_legacy registered [ ***] A start job is running for Configur…only root support (11s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg224-server login: [ 90.589118] libcfs: loading out-of-tree module taints kernel. [ 90.627927] Key type ._llcrypt registered [ 90.635705] Key type .llcrypt registered [ 90.777102] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_hostid [ 110.370868] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 112.111647] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 112.129706] alg: No test for adler32 (adler32-zlib) [ 113.602445] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 114.764218] hrtimer: interrupt took 3261981 ns [ 114.877075] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [ 116.625469] Key type lgssc registered [ 118.387054] Lustre: Echo OBD driver; http://www.lustre.org/ [ 133.389371] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 180.345250] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 197.568242] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 197.596937] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 199.060530] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 199.143818] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 199.322859] Lustre: lustre-MDT0000: new disk, initializing [ 199.487233] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 199.506241] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 205.894257] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 222.704929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 222.825452] Lustre: 6512:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 222.857983] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 222.865573] Lustre: Skipped 1 previous similar message [ 222.957228] Lustre: lustre-MDT0001: new disk, initializing [ 223.084686] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 223.148973] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 223.187212] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 228.848956] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 233.966167] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 245.424462] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 245.832654] Lustre: lustre-OST0000: new disk, initializing [ 245.836837] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 245.846780] Lustre: 8453:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 245.921033] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 251.428524] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 251.442761] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 251.533504] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 252.515518] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 269.612985] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 269.845498] Lustre: lustre-OST0001: new disk, initializing [ 269.858156] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 269.870448] Lustre: 9526:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 270.018875] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 276.563082] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 276.571925] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 276.665056] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 277.977037] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 291.092566] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 299.718828] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 308.327865] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing check_logdir /tmp/testlogs/ [ 314.790746] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing yml_node [ 320.760703] Lustre: DEBUG MARKER: Client: 2.17.58.2 [ 324.109242] Lustre: DEBUG MARKER: MDS: 2.17.58.2 [ 327.809458] Lustre: DEBUG MARKER: OSS: 2.17.58.2 [ 329.964264] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 3 08:26:43 EDT 2026 [ 353.493314] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 355.877108] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 376.804335] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 376.810149] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 376.841237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 377.316822] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 377.331882] Lustre: Skipped 2 previous similar messages [ 379.146308] Lustre: server umount lustre-MDT0000 complete [ 382.947291] LustreError: 6505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788438458 with bad export cookie 15017667546065715404 [ 382.956214] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 382.959108] LustreError: 6505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 387.553217] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 387.556263] LustreError: 6521:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 387.556433] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 387.556438] Lustre: Skipped 4 previous similar messages [ 387.565380] Lustre: Skipped 1 previous similar message [ 387.586286] LustreError: 6521:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 392.679083] LustreError: 6520:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 392.680511] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 392.693177] LustreError: 6520:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 392.696374] Lustre: Skipped 2 previous similar messages [ 393.697397] LustreError: 6521:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 393.704904] LustreError: 6521:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 397.280231] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 397.532996] Lustre: server umount lustre-MDT0001 complete [ 402.512595] Lustre: server umount lustre-OST0000 complete [ 407.173900] Lustre: server umount lustre-OST0001 complete [ 415.222993] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_hostid [ 425.831260] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 469.392913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 469.748295] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 469.777628] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 469.834691] Lustre: lustre-MDT0000: new disk, initializing [ 469.920567] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 469.956731] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 476.435973] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 491.206451] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 491.513927] Lustre: 15083:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 491.733348] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 491.738797] Lustre: Skipped 1 previous similar message [ 491.876405] Lustre: lustre-MDT0001: new disk, initializing [ 492.068895] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 492.193677] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 492.222593] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 498.389305] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 510.058643] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 516.384943] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 525.099463] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 525.458270] Lustre: lustre-OST0000: new disk, initializing [ 525.464138] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 525.471791] Lustre: 17054:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 525.606267] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 527.445123] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 527.455053] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 527.613220] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 533.473770] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 544.678505] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 550.728922] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 550.951772] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 556.113305] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 556.350702] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 558.561503] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 558.565474] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 558.576551] LustreError: Skipped 1 previous similar message [ 558.580521] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 558.580530] Lustre: Skipped 2 previous similar messages [ 558.625821] Lustre: Skipped 1 previous similar message [ 563.693135] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 563.702733] Lustre: Skipped 2 previous similar messages [ 564.829785] Lustre: server umount lustre-OST0000 complete [ 573.925308] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 573.945945] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 573.958260] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 574.004308] Lustre: Skipped 1 previous similar message [ 575.755627] Lustre: server umount lustre-MDT0000 complete [ 579.045376] LustreError: 15096:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 579.072626] LustreError: 15096:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 580.850299] LustreError: 15075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788438655 with bad export cookie 15017667546065716937 [ 580.860749] LustreError: 15075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 580.863267] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 581.394399] Lustre: server umount lustre-MDT0001 complete [ 589.027480] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 08:31:02 (1788438662) [ 612.720575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 613.236478] LustreError: 19930:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 613.383450] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 619.243253] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 623.591902] LustreError: 19931:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 623.617212] LustreError: 19931:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 629.883924] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 635.674546] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 644.876419] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 651.208099] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 653.249889] Lustre: Failing over lustre-MDT0000 [ 653.625833] Lustre: server umount lustre-MDT0000 complete [ 655.331834] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 655.346557] Lustre: Skipped 1 previous similar message [ 655.350678] LustreError: 19926:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 655.362945] LustreError: 19926:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 671.525124] Lustre: 3642:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788438730/real 1788438730] req@ffff9e82810cb800 x1875313363690112/t0(0) o400->MGC192.168.202.124@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788438746 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 671.542519] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 673.909791] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 681.825969] LustreError: 3641:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9e82810c9f80 x1875313363693056/t0(0) o250->MGC192.168.202.124@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 682.183689] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 682.191611] Lustre: Skipped 1 previous similar message [ 682.236182] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 687.581896] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 687.585937] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 687.587204] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 687.628695] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 698.526447] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 703.065178] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 703.093169] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 703.096035] Lustre: Skipped 1 previous similar message [ 709.606847] Lustre: server umount lustre-MDT0000 complete [ 713.185884] LustreError: 19930:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 713.223523] LustreError: 19930:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 15 previous similar messages [ 714.382247] LustreError: 19912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788438789 with bad export cookie 15017667546065718351 [ 714.398497] LustreError: 19912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 714.409225] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 715.104431] Lustre: server umount lustre-MDT0001 complete [ 727.206463] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 733.520980] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 744.644672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 751.051447] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 760.248546] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 767.204426] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 769.910527] Lustre: Failing over lustre-MDT0000 [ 770.084080] Lustre: server umount lustre-MDT0000 complete [ 770.991209] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 771.005564] Lustre: Skipped 1 previous similar message [ 779.235016] LustreError: 23271:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 779.257078] LustreError: 23271:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 787.424406] Lustre: 3643:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788438846/real 1788438846] req@ffff9e8282773480 x1875313363721088/t0(0) o400->MGC192.168.202.124@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788438862 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 787.465554] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 792.649959] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 798.689190] LustreError: 3641:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9e83bf3e6d80 x1875313363724288/t0(0) o250->MGC192.168.202.124@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 799.107116] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 804.329780] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 804.347333] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 804.359304] Lustre: Skipped 1 previous similar message [ 804.404749] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 804.777474] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 816.020707] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 821.734509] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 821.742716] Lustre: Skipped 2 previous similar messages [ 826.853626] Lustre: server umount lustre-MDT0000 complete [ 831.827379] LustreError: 23252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788438906 with bad export cookie 15017667546065719289 [ 831.834872] LustreError: 23252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 831.845595] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 832.425741] Lustre: server umount lustre-MDT0001 complete [ 843.911727] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 08:35:17 (1788438917) [ 864.284858] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_hostid [ 877.054077] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 928.917395] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 942.441335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 942.883352] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 942.988864] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 943.148297] Lustre: lustre-MDT0000: new disk, initializing [ 943.218819] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 943.224173] Lustre: Skipped 3 previous similar messages [ 943.235893] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 949.636828] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 963.431599] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 963.573980] Lustre: 29653:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 963.967474] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 969.409376] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 981.082522] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 988.246174] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 997.377681] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 997.853368] Lustre: 31625:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 999.890306] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 999.950323] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 1000.097617] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1000.138235] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 1005.413561] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 1016.789074] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1023.434875] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 1051.106815] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1051.119086] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1051.130023] Lustre: Skipped 3 previous similar messages [ 1051.140861] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1051.158900] Lustre: Skipped 2 previous similar messages [ 1062.368210] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1062.699269] Lustre: server umount lustre-OST0000 complete [ 1071.600305] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1073.578707] Lustre: server umount lustre-MDT0000 complete [ 1076.708917] LustreError: 30529:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1076.736430] LustreError: 30529:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 15 previous similar messages [ 1077.599957] LustreError: 29645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439152 with bad export cookie 15017667546065719695 [ 1077.605753] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1077.613376] LustreError: 29645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1078.083159] Lustre: server umount lustre-MDT0001 complete [ 1091.300200] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [ 1095.001856] Key type lgssc unregistered [ 1095.406845] LNet: 33603:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1095.420494] LNetError: 33603:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1095.447485] LNet: Removed LNI 192.168.202.124@tcp [ 1096.691274] Key type .llcrypt unregistered [ 1096.697139] Key type ._llcrypt unregistered [ 1124.955355] Key type ._llcrypt registered [ 1124.957434] Key type .llcrypt registered [ 1125.059773] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 1126.594680] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1126.698358] alg: No test for adler32 (adler32-zlib) [ 1127.902549] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1128.124817] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [ 1129.866855] Key type lgssc registered [ 1131.141812] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1140.932485] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1140.949390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1142.376776] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1146.523180] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 1151.972945] LustreError: 35014:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1151.982358] LustreError: 35014:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1156.144929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1156.739250] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1162.008677] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 1170.709662] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1177.820923] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1186.996945] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1187.468283] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1196.955750] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 1201.868710] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1206.570533] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1221.524652] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 08:41:35 (1788439295) [ 1224.319987] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1226.512900] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1235.031482] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 08:41:48 (1788439308) [ 1246.375465] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 08:41:59 (1788439319) [ 1248.096031] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1261.067074] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1261.074025] Lustre: Skipped 2 previous similar messages [ 1263.761172] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 08:42:16 (1788439336) [ 1265.741203] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1265.745769] Lustre: Skipped 2 previous similar messages [ 1274.333603] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1274.342939] Lustre: Skipped 4 previous similar messages [ 1292.322227] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 08:42:45 (1788439365) [ 1295.948848] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1300.536503] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1310.146554] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 08:43:03 (1788439383) [ 1312.632257] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1312.640168] Lustre: Skipped 6 previous similar messages [ 1329.982779] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 08:43:23 (1788439403) [ 1347.558687] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1347.568634] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1347.586682] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1350.626867] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1350.637810] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1350.648727] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1355.747887] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1355.750143] Lustre: Skipped 2 previous similar messages [ 1360.353221] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1360.686487] Lustre: server umount lustre-OST0000 complete [ 1365.987982] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1365.993886] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1366.012270] Lustre: Skipped 1 previous similar message [ 1366.041592] Lustre: Skipped 1 previous similar message [ 1371.107504] LustreError: 35015:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1371.247827] Lustre: server umount lustre-MDT0000 complete [ 1375.707271] LustreError: 39438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439450 with bad export cookie 15714274303566352355 [ 1375.708940] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1375.728306] LustreError: 39438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1376.475456] Lustre: server umount lustre-MDT0001 complete [ 1390.082243] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [ 1393.068031] Key type lgssc unregistered [ 1393.299956] LNet: 42874:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1393.313045] LNetError: 42874:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1393.339338] LNet: Removed LNI 192.168.202.124@tcp [ 1394.135211] Key type .llcrypt unregistered [ 1394.138826] Key type ._llcrypt unregistered [ 1418.138849] Key type ._llcrypt registered [ 1418.140534] Key type .llcrypt registered [ 1418.345098] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_hostid [ 1440.472383] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 1442.005124] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1442.297449] alg: No test for adler32 (adler32-zlib) [ 1443.418822] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1443.821161] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [ 1445.560312] Key type lgssc registered [ 1447.022271] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1492.786378] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1492.848163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1494.149638] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1494.185491] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1494.310244] Lustre: lustre-MDT0000: new disk, initializing [ 1494.403796] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1494.424055] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1500.123668] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 1512.899934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1513.091319] Lustre: 46029:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1513.161556] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1513.164948] Lustre: Skipped 1 previous similar message [ 1513.238973] Lustre: lustre-MDT0001: new disk, initializing [ 1513.369416] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1513.447172] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1513.459484] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1519.355537] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 1530.964776] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1537.297983] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1545.550876] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1545.803394] Lustre: lustre-OST0000: new disk, initializing [ 1545.810748] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1545.822988] Lustre: 47997:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1545.938080] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1554.131750] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 1554.612862] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1554.620663] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1554.799482] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1565.571493] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1572.451160] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1572.781782] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1578.967661] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1579.287270] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1585.635192] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1585.646187] Lustre: Skipped 1 previous similar message [ 1585.648795] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1585.649040] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1585.659165] Lustre: Skipped 1 previous similar message [ 1588.062571] Lustre: server umount lustre-OST0000 complete [ 1595.373119] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1595.383882] LustreError: Skipped 1 previous similar message [ 1595.395435] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1595.421937] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1601.003569] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1601.014505] Lustre: Skipped 2 previous similar messages [ 1605.118485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1605.147164] Lustre: Skipped 1 previous similar message [ 1608.164629] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1608.908535] Lustre: server umount lustre-MDT0000 complete [ 1614.642708] LustreError: 46021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439689 with bad export cookie 15661463457597962395 [ 1614.656854] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1614.669851] LustreError: 46021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1615.163640] Lustre: server umount lustre-MDT0001 complete [ 1628.880121] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 08:48:21 (1788439701) [ 1638.152502] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1638.994719] LustreError: 50461:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1639.021894] LustreError: 50461:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1639.119724] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1644.515576] LustreError: 50462:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1644.527560] LustreError: 50462:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1646.343264] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 1649.636859] LustreError: 50461:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1654.758625] LustreError: 50462:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1657.000118] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1657.464195] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1664.067824] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 1674.428333] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1682.240668] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1690.648141] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1690.908890] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1698.623492] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 1708.957227] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1713.927974] LustreError: 52993:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1715.382741] Lustre: Modifying parameter general.timeout=1 in log params [ 1716.701752] Lustre: Modifying parameter general.timeout=2 in log params [ 1719.948956] Lustre: Modifying parameter general.timeout=5 in log params [ 1719.953765] Lustre: Skipped 2 previous similar messages [ 1724.296425] Lustre: Modifying parameter general.timeout=9 in log params [ 1724.298783] Lustre: Skipped 3 previous similar messages [ 1732.942885] Lustre: Modifying parameter general.timeout=17 in log params [ 1732.946834] Lustre: Skipped 7 previous similar messages [ 1749.736731] Lustre: Modifying parameter general.timeout=35 in log params [ 1749.743119] Lustre: Skipped 17 previous similar messages [ 1782.152046] Lustre: Modifying parameter general.timeout=66 in log params [ 1782.155011] Lustre: Skipped 30 previous similar messages [ 1846.327306] Lustre: Modifying parameter general.timeout=125 in log params [ 1846.336623] Lustre: Skipped 58 previous similar messages [ 1864.219207] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 08:52:17 (1788439937) [ 1868.061191] Lustre: 59950:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1871.210377] Lustre: 60057:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1874.451211] Lustre: 60162:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1879.016656] Lustre: 60358:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=50 (mode = 2) failed: rc = -17 [ 1879.046943] Lustre: 60358:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1914.179225] Lustre: 61830:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1914.186962] Lustre: 61830:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1914.213039] LustreError: 61830:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1920.072612] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 08:53:13 (1788439993) [ 1935.843945] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1935.848602] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1935.892702] Lustre: Skipped 3 previous similar messages [ 1935.905969] Lustre: Skipped 2 previous similar messages [ 1940.966881] LustreError: 50461:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1940.990971] LustreError: 50461:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1941.098672] Lustre: server umount lustre-MDT0000 complete [ 1946.056984] LustreError: 50442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440021 with bad export cookie 15661463457597963277 [ 1946.061375] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1946.067906] LustreError: 50442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1946.102158] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1946.128373] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1946.319791] Lustre: server umount lustre-MDT0001 complete [ 1952.723502] Lustre: server umount lustre-OST0000 complete [ 1982.822368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1983.155753] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1983.190400] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1983.197176] Lustre: Skipped 27 previous similar messages [ 1983.388777] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1988.812874] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 1999.558501] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1999.736182] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1999.767410] Lustre: 64363:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2005.434978] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 2014.318266] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2021.533168] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2029.759234] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2029.974325] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2030.399411] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2030.412298] Lustre: Skipped 1 previous similar message [ 2038.458497] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 2051.809971] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2055.701348] Lustre: 66932:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 2066.403297] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2066.410507] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2066.419864] LustreError: Skipped 1 previous similar message [ 2066.449596] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2066.479261] Lustre: Skipped 1 previous similar message [ 2072.326275] Lustre: server umount lustre-OST0000 complete [ 2076.642179] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2083.238162] Lustre: server umount lustre-MDT0000 complete [ 2086.882167] LustreError: 64374:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2086.908837] LustreError: 64374:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2087.702464] LustreError: 65105:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440162 with bad export cookie 15661463457597966546 [ 2087.707380] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2087.714708] LustreError: 65105:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2088.440090] Lustre: server umount lustre-MDT0001 complete [ 2100.264890] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [ 2103.157644] Key type lgssc unregistered [ 2103.535288] LNet: 68142:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2103.550796] LNetError: 68142:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2104.619315] LNet: Removed LNI 192.168.202.124@tcp [ 2105.557442] Key type .llcrypt unregistered [ 2105.559368] Key type ._llcrypt unregistered [ 2133.002502] Key type ._llcrypt registered [ 2133.004915] Key type .llcrypt registered [ 2133.103632] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 2135.007065] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2135.077642] alg: No test for adler32 (adler32-zlib) [ 2136.244384] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2136.547591] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [ 2138.296182] Key type lgssc registered [ 2139.621100] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2149.568888] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2149.601851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2151.229182] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2156.351948] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 2161.634706] LustreError: 69556:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2161.665336] LustreError: 69556:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2166.754914] LustreError: 69555:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2167.170043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2168.319418] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2174.185302] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 2185.236132] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2192.754823] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2201.173216] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2202.005062] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2209.807854] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 2219.112654] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2232.453844] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2242.026036] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2242.032414] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2242.058888] Lustre: Skipped 2 previous similar messages [ 2242.070652] Lustre: Skipped 2 previous similar messages [ 2247.142288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2248.219716] Lustre: server umount lustre-MDT0000 complete [ 2252.267401] LustreError: 69556:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2252.287745] LustreError: 69556:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2252.363860] LustreError: 71409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440327 with bad export cookie 3962016549027327238 [ 2252.368879] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2252.374386] LustreError: 71409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2252.927827] Lustre: server umount lustre-MDT0001 complete [ 2267.944198] Lustre: server umount lustre-OST0000 complete [ 2300.893184] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2301.426568] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2301.494198] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2301.889917] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2307.460059] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 2317.195201] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2317.336232] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2317.366772] Lustre: 74431:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2318.009555] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2323.252789] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 2331.380674] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2337.923635] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2345.952636] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2346.142626] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2346.504653] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2354.028610] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 2363.573425] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2365.786936] Lustre: 77006:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2368.038389] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2373.601647] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2373.613522] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2373.623500] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2373.627867] Lustre: Skipped 2 previous similar messages [ 2374.127419] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2379.247782] LustreError: 76284:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2379.415829] Lustre: server umount lustre-OST0000 complete [ 2384.353701] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2384.362725] Lustre: Skipped 1 previous similar message [ 2384.368903] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2384.374785] Lustre: Skipped 2 previous similar messages [ 2389.475305] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2389.489036] Lustre: Skipped 1 previous similar message [ 2390.155558] Lustre: server umount lustre-MDT0000 complete [ 2394.546814] LustreError: 77008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440469 with bad export cookie 3962016549027328085 [ 2394.552650] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2394.556468] LustreError: 77008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2395.125378] Lustre: server umount lustre-MDT0001 complete [ 2407.596951] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [ 2411.453547] Key type lgssc unregistered [ 2411.878189] LNet: 78231:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2411.899853] LNetError: 78231:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2411.927651] LNet: Removed LNI 192.168.202.124@tcp [ 2412.864749] Key type .llcrypt unregistered [ 2412.868313] Key type ._llcrypt unregistered [ 2438.355207] Key type ._llcrypt registered [ 2438.358439] Key type .llcrypt registered [ 2438.548433] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 2440.332526] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2440.351978] alg: No test for adler32 (adler32-zlib) [ 2441.614135] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2441.888501] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [ 2443.648404] Key type lgssc registered [ 2445.283965] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2454.854352] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2454.881492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2456.622686] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2461.840510] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 2466.794116] LustreError: 79642:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2466.816193] LustreError: 79642:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2470.593771] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2471.343534] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2475.395737] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 2482.891194] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2490.132949] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2497.961727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2498.476052] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2506.174647] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 2515.330595] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2522.479499] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2523.529432] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2524.765270] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2526.933945] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2526.938604] Lustre: Skipped 1 previous similar message [ 2531.312769] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2531.319768] Lustre: Skipped 3 previous similar messages [ 2540.591764] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2540.595661] Lustre: Skipped 7 previous similar messages [ 2556.848933] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2556.859549] Lustre: Skipped 14 previous similar messages [ 2589.511072] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2589.527936] Lustre: Skipped 29 previous similar messages [ 2643.119968] Lustre: 87757:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2646.591980] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 09:05:19 (1788440719) [ 2648.377682] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2650.789862] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 09:05:24 (1788440724) [ 2655.201121] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2655.209987] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2655.230850] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2659.302636] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2659.313245] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2659.315634] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2659.317353] Lustre: Skipped 1 previous similar message [ 2661.542285] Lustre: server umount lustre-OST0000 complete [ 2669.544713] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2669.562238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2669.572550] Lustre: Skipped 1 previous similar message [ 2674.657428] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2674.669696] Lustre: Skipped 1 previous similar message [ 2679.777552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2679.784717] Lustre: Skipped 1 previous similar message [ 2680.288309] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2680.518449] Lustre: server umount lustre-MDT0000 complete [ 2684.898864] LustreError: 79642:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2684.916924] LustreError: 79642:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2685.249780] LustreError: 79622:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440760 with bad export cookie 15883140807685403189 [ 2685.254199] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2685.263640] LustreError: 79622:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2685.518694] Lustre: server umount lustre-MDT0001 complete [ 2698.968717] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [ 2702.183291] Key type lgssc unregistered [ 2702.517713] LNet: 89007:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2702.534898] LNetError: 89007:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2702.570061] LNet: Removed LNI 192.168.202.124@tcp [ 2703.589550] Key type .llcrypt unregistered [ 2703.592834] Key type ._llcrypt unregistered [ 2709.022565] Key type ._llcrypt registered [ 2709.024334] Key type .llcrypt registered [ 2709.192619] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2716.149262] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules [ 2717.699554] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2717.765542] alg: No test for adler32 (adler32-zlib) [ 2719.027435] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2719.338696] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [ 2721.098709] Key type lgssc registered [ 2722.892633] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2725.211904] LustreError: 90266:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2744.766669] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 2747.234992] LustreError: 90266:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2754.124243] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2754.163475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2755.744780] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2760.059301] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 2765.289179] LustreError: 91274:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2765.316327] LustreError: 91274:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2770.426588] LustreError: 91275:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2770.469762] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 09:07:23 (1788440843) [ 2775.522371] LustreError: 91274:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2775.571327] LustreError: 91306:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2775.585529] LustreError: 91306:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2775.593219] LustreError: 91306:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 20, retries 0, failed: rc = -5 [ 2775.886904] Lustre: server umount lustre-MDT0000 complete [ 2793.378752] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [ 2796.653443] Key type lgssc unregistered [ 2796.937991] LNet: 92800:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2796.945236] LNetError: 92800:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2796.962766] LNet: Removed LNI 192.168.202.124@tcp [ 2797.922106] Key type .llcrypt unregistered [ 2797.928736] Key type ._llcrypt unregistered [ 2818.744331] Key type ._llcrypt registered [ 2818.748385] Key type .llcrypt registered [ 2818.878623] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 2820.360720] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2820.423074] alg: No test for adler32 (adler32-zlib) [ 2821.700511] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2822.061889] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [ 2823.720560] Key type lgssc registered [ 2825.070712] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2834.173137] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2834.220633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2836.113570] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2840.419269] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 2846.692023] LustreError: 94015:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2846.706915] LustreError: 94015:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2850.190525] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2850.745608] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2855.400456] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 2864.571482] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2871.525424] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2879.914107] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2880.695489] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2888.243338] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 2897.434706] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2948.535875] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 09:10:22 (1788441022) [ 2950.694663] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2953.055961] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 09:10:26 (1788441026) [ 2961.900936] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2961.909409] Lustre: Skipped 2 previous similar messages [ 2961.915330] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2961.934318] Lustre: Skipped 2 previous similar messages [ 2966.577206] Lustre: server umount lustre-MDT0000 complete [ 2967.025607] LustreError: 94016:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2967.044513] LustreError: 94016:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2971.259337] LustreError: 93997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441046 with bad export cookie 10001467442304195782 [ 2971.271967] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2971.277758] LustreError: 93997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2971.635994] Lustre: server umount lustre-MDT0001 complete [ 2987.065398] Lustre: server umount lustre-OST0000 complete [ 3002.201732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3002.920901] LustreError: 98291:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3003.113637] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3008.248672] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 3008.486848] LustreError: 98291:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3008.508995] LustreError: 98291:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3013.603354] LustreError: 98292:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3017.732415] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3018.467510] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3023.718979] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 3034.696581] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3043.709672] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3062.383991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3062.596884] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 3062.602443] LustreError: 98280:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 3062.622638] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 3062.628724] LustreError: 100377:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -98 [ 3062.632967] LustreError: 100377:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 3062.637854] LustreError: 100377:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 3062.711512] Lustre: server umount lustre-OST0000 complete [ 3062.713934] LustreError: 100377:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 3074.631201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3075.030646] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 3075.041223] LustreError: 98281:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 3075.072558] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 3075.081913] LustreError: 100842:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -98 [ 3075.088940] LustreError: 100842:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 3075.099201] LustreError: 100842:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 3075.220913] Lustre: server umount lustre-OST0000 complete [ 3075.225055] LustreError: 100842:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 3087.947919] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3088.274957] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3088.288870] Lustre: Found index 0 for lustre-OST0000, updating log [ 3088.296636] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 3088.358550] LustreError: 101360:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3088.428501] Lustre: lustre-OST0000: new disk, initializing [ 3088.431817] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3088.442671] Lustre: 101377:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3088.886643] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3098.013727] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 3102.339964] Lustre: Failing over lustre-OST0000 [ 3102.477756] Lustre: server umount lustre-OST0000 complete [ 3108.329754] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3108.334873] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3108.343849] Lustre: Skipped 1 previous similar message [ 3108.351016] Lustre: Skipped 1 previous similar message [ 3113.442157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3113.455573] Lustre: Skipped 1 previous similar message [ 3117.610587] Lustre: server umount lustre-MDT0000 complete [ 3118.562122] LustreError: 98287:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3118.579765] LustreError: 98287:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3119.587315] Lustre: 93326:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788441178/real 1788441178] req@0000000068df31cf x1875316202118144/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788441194 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3119.631804] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3120.608423] Lustre: 93324:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788441179/real 1788441179] req@00000000428da6d3 x1875316202118528/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788441195 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3122.116281] LustreError: 98272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441197 with bad export cookie 10001467442304198526 [ 3122.121531] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3122.142718] LustreError: 98272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3122.900262] Lustre: server umount lustre-MDT0001 complete [ 3132.155560] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 09:13:26 (1788441206) [ 3149.828591] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 3160.567211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3160.935360] LustreError: 103726:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3160.953581] LustreError: 103726:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3161.038065] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3166.726065] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 3176.395966] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3183.311480] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 3186.808727] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 3194.937238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3195.677951] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3195.680519] Lustre: Skipped 1 previous similar message [ 3195.694938] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3202.872349] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3203.085989] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 3206.018602] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3206.019053] Lustre: lustre-OST0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 3206.044595] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 3213.244507] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3213.424634] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3213.566234] Lustre: lustre-OST0001: new disk, initializing [ 3213.571358] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3213.579283] Lustre: 106047:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3221.185541] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3221.196001] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3221.350441] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3222.077778] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 3229.902536] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3237.636304] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3244.001130] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 3244.008485] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3244.033250] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3248.619335] Lustre: server umount lustre-MDT0001 complete [ 3250.754112] LustreError: 105240:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3250.792359] LustreError: 105240:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3259.788399] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3259.912685] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3259.918940] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3259.989659] Lustre: 103716:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3260.683885] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3260.698074] Lustre: Skipped 1 previous similar message [ 3261.090253] LustreError: 107578:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 3261.096610] LustreError: 107578:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 3261.117672] LustreError: 103718:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 3261.244304] LustreError: 107592:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.124@tcp: cfg command failed: rc = -17 [ 3261.256128] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3261.256128] [ 3266.047425] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3266.057494] Lustre: Skipped 1 previous similar message [ 3266.436805] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 3278.520504] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 09:15:51 (1788441351) [ 3302.702530] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 3324.760447] Lustre: 109694:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3350.226668] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3354.159185] Lustre: 110829:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3449.826518] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3449.836468] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3449.846197] Lustre: Skipped 2 previous similar messages [ 3449.860624] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3449.872023] Lustre: Skipped 4 previous similar messages [ 3452.732542] Lustre: server umount lustre-MDT0000 complete [ 3455.457290] LustreError: 106267:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3455.476817] LustreError: 106267:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 3456.876110] LustreError: 103707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441532 with bad export cookie 10001467442304199534 [ 3456.877354] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3456.882971] LustreError: 103707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3457.569167] Lustre: server umount lustre-MDT0001 complete [ 3471.133891] Lustre: server umount lustre-OST0000 complete [ 3485.917066] Lustre: server umount lustre-OST0001 complete [ 3493.417022] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3504.544716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3515.715033] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3527.057773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3549.323729] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 3560.804365] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3560.836804] Lustre: lustre-MDT0000: reset Object Index mappings [ 3561.640346] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3566.733555] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 3576.705180] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3576.739240] Lustre: lustre-MDT0001: reset Object Index mappings [ 3582.624778] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 3586.332344] Lustre: 115938:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3593.675220] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3601.548428] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 3602.490079] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:227 to 0x280000400:257) [ 3606.589159] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:387 to 0x280000401:418) [ 3612.362393] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3618.297814] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:161 to 0x2c0000401:193) [ 3618.315405] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 3621.596764] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 3630.390898] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3634.717596] Lustre: 117813:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3691.830805] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 09:22:45 (1788441765) [ 3720.681518] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3720.683692] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3720.686254] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3720.686264] Lustre: Skipped 4 previous similar messages [ 3720.715171] Lustre: Skipped 4 previous similar messages [ 3723.667830] Lustre: server umount lustre-MDT0000 complete [ 3725.813844] LustreError: 115526:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3725.839211] LustreError: 115526:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 3728.283806] LustreError: 114770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441803 with bad export cookie 10001467442304246651 [ 3728.307457] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3728.311816] LustreError: 114770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3735.774966] Lustre: server umount lustre-MDT0001 complete [ 3740.791723] LustreError: 118590:0:(ldlm_resource.c:1207:ldlm_resource_complain()) filter-lustre-OST0000_UUID: namespace resource [0x280000401:0x133:0x0].0x0 (ffff9e83bcfc1c00) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 3746.963736] Lustre: server umount lustre-OST0000 complete [ 3757.749295] Lustre: server umount lustre-OST0001 complete [ 3766.242627] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_hostid [ 3778.559746] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 3828.565599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3828.910522] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3828.961945] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3829.085977] Lustre: lustre-MDT0000: new disk, initializing [ 3829.198891] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3829.209494] Lustre: Skipped 3 previous similar messages [ 3829.239744] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3834.153771] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 3842.368939] Lustre: server umount lustre-MDT0000 complete [ 3852.827292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3853.212074] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3853.244706] Lustre: 122102:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3853.262951] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3858.625274] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 3863.977718] Lustre: server umount lustre-MDT0000 complete [ 3874.077597] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 09:25:47 (1788441947) [ 3876.018351] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3878.133786] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 09:25:51 (1788441951) [ 3899.933554] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 3913.147933] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3918.822406] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 3928.670627] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3928.795927] Lustre: 124152:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3928.861567] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3928.867727] Lustre: Skipped 1 previous similar message [ 3928.972284] Lustre: lustre-MDT0001: new disk, initializing [ 3929.219488] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3929.234253] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3934.046285] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 3939.416196] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3946.161798] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3946.688403] Lustre: lustre-OST0000: new disk, initializing [ 3946.695586] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3946.724169] Lustre: 125731:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3947.950665] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3947.955874] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3948.133876] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3953.977951] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 3965.172978] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3965.360547] Lustre: lustre-OST0001: new disk, initializing [ 3965.364751] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3965.371250] Lustre: 126602:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3967.100697] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3967.112262] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3967.242282] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3972.389443] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 3982.479807] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3987.206918] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4405.782312] Lustre: 133024:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4405.797624] Lustre: 133024:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4405.807439] Lustre: 133024:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4405.813606] Lustre: 133024:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4405.826648] Lustre: 133024:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4405.835449] Lustre: 133024:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4678.592954] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 09:39:11 (1788442751) [ 4697.569808] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4697.572087] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4697.586259] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4697.586268] Lustre: Skipped 7 previous similar messages [ 4697.614472] Lustre: Skipped 7 previous similar messages [ 4701.071243] Lustre: server umount lustre-MDT0000 complete [ 4702.693052] LustreError: 126611:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4702.722612] LustreError: 126611:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 4705.094135] LustreError: 124146:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442780 with bad export cookie 10001467442304316021 [ 4705.095915] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4705.114935] LustreError: 124146:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4707.810874] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4707.822239] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4707.833457] Lustre: Skipped 1 previous similar message [ 4707.845474] Lustre: Skipped 4 previous similar messages [ 4711.890709] Lustre: server umount lustre-MDT0001 complete [ 4730.336128] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4730.470585] Lustre: server umount lustre-OST0000 complete [ 4740.927340] Lustre: server umount lustre-OST0001 complete [ 4748.419616] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_hostid [ 4760.216954] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 4801.408514] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4801.733489] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4801.775578] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4801.886625] Lustre: lustre-MDT0000: new disk, initializing [ 4801.993108] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4801.997463] Lustre: Skipped 5 previous similar messages [ 4802.008901] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4807.682343] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 4818.268544] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4818.446253] Lustre: 136074:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4818.526669] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4818.531696] Lustre: Skipped 1 previous similar message [ 4818.706410] Lustre: lustre-MDT0001: new disk, initializing [ 4818.890756] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4818.903301] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4823.178404] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 4833.681828] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4840.336280] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4847.767058] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4848.132516] Lustre: lustre-OST0000: new disk, initializing [ 4848.136718] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4848.152514] Lustre: 138044:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4849.441392] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4849.460372] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4849.640458] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4855.840885] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 4866.589845] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4877.151766] Lustre: lustre-MDD0000: changelog on [ 4880.259573] Lustre: lustre-MDD0001: changelog on [ 6904.579499] Lustre: lustre-MDD0001: changelog off [ 6906.655429] Lustre: 136082:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6907.716422] Lustre: lustre-MDD0000: changelog off [ 6918.123973] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6918.128599] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6918.149628] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6918.149636] Lustre: Skipped 2 previous similar messages [ 6923.254241] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6923.260881] Lustre: Skipped 2 previous similar messages [ 6928.352181] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6928.358238] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6928.372630] Lustre: Skipped 1 previous similar message [ 6928.539589] Lustre: server umount lustre-OST0000 complete [ 6933.480657] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6933.501330] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6933.506522] Lustre: Skipped 1 previous similar message [ 6933.507485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6933.522335] LustreError: Skipped 1 previous similar message [ 6938.594382] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6938.608305] Lustre: Skipped 2 previous similar messages [ 6939.533550] Lustre: server umount lustre-MDT0000 complete [ 6943.659235] LustreError: 136068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445018 with bad export cookie 10001467442305089941 [ 6943.670589] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6943.679752] LustreError: 136068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6944.256635] Lustre: server umount lustre-MDT0001 complete [ 6956.136699] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [ 6959.364747] Key type lgssc unregistered [ 6959.833588] LNet: 147209:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6959.850913] LNetError: 147209:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6959.883076] LNet: Removed LNI 192.168.202.124@tcp [ 6961.016165] Key type .llcrypt unregistered [ 6961.020571] Key type ._llcrypt unregistered [ 6982.071677] Key type ._llcrypt registered [ 6982.074650] Key type .llcrypt registered [ 6982.173090] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_hostid [ 6998.323878] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 6999.238526] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6999.388136] alg: No test for adler32 (adler32-zlib) [ 7000.454923] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 7000.683460] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [ 7002.464172] Key type lgssc registered [ 7003.622095] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7036.710114] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7036.743518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7038.075351] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7038.103446] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7038.167982] Lustre: lustre-MDT0000: new disk, initializing [ 7038.224954] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7038.237379] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7043.426253] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 7054.942434] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7055.063338] Lustre: 150404:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7055.120932] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7055.125244] Lustre: Skipped 1 previous similar message [ 7055.188368] Lustre: lustre-MDT0001: new disk, initializing [ 7055.274468] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7055.327174] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7055.340936] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7061.553196] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 7074.591092] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7081.200904] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7087.993519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7088.220973] Lustre: lustre-OST0000: new disk, initializing [ 7088.224836] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7088.230884] Lustre: 152375:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7088.332579] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7093.863867] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7093.886943] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7094.044069] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7094.335594] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 7103.535871] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7109.956157] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_module obdecho/obdecho [ 7111.644850] LustreError: 153377:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 7111.660095] LustreError: 153377:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 7113.482985] LustreError: 153474:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 7124.963522] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7124.968973] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7124.982493] Lustre: Skipped 2 previous similar messages [ 7124.989018] Lustre: Skipped 1 previous similar message [ 7130.092676] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7130.106559] Lustre: Skipped 3 previous similar messages [ 7130.290648] Lustre: server umount lustre-MDT0000 complete [ 7134.517137] LustreError: 150398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445209 with bad export cookie 471804300646978077 [ 7134.528537] LustreError: 150398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7134.529533] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7134.911524] Lustre: server umount lustre-MDT0001 complete [ 7148.161924] Lustre: server umount lustre-OST0000 complete [ 7172.271499] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_hostid [ 7186.285574] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 7225.576092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7225.767689] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7225.789353] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7225.869830] Lustre: lustre-MDT0000: new disk, initializing [ 7226.004827] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7226.052938] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7231.142158] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 7240.822705] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7240.980229] Lustre: 157191:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7241.049880] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7241.060199] Lustre: Skipped 1 previous similar message [ 7241.146823] Lustre: lustre-MDT0001: new disk, initializing [ 7241.245859] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7241.289479] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7241.312516] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7245.718915] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 7254.361752] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7260.521611] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7267.983535] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7268.302573] Lustre: lustre-OST0000: new disk, initializing [ 7268.314447] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7268.326438] Lustre: 159159:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7268.425852] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7274.068898] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7274.074584] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7274.340282] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7274.504406] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 7285.474830] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7291.504634] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7291.824638] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7297.601550] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7297.879042] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7305.206368] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7305.209147] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7305.217358] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7305.224851] LustreError: Skipped 1 previous similar message [ 7305.261777] Lustre: Skipped 1 previous similar message [ 7306.723409] Lustre: server umount lustre-OST0000 complete [ 7312.869204] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7312.873632] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7312.882993] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7312.885881] Lustre: Skipped 1 previous similar message [ 7315.444795] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7316.836646] Lustre: server umount lustre-MDT0000 complete [ 7320.546406] LustreError: 157203:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7320.565316] LustreError: 157203:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7320.964532] LustreError: 157183:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445396 with bad export cookie 471804300646979036 [ 7320.968022] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7320.972928] LustreError: 157183:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7321.379979] Lustre: server umount lustre-MDT0001 complete [ 7324.895984] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 10:23:18 (1788445398) [ 7326.548552] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 7328.746600] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 10:23:22 (1788445402) [ 7336.858307] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7337.522969] LustreError: 161571:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7337.642568] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7342.633204] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 7343.073418] LustreError: 161572:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7347.182295] LustreError: 161571:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7350.947155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7356.388538] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 7365.292412] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7375.008455] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7384.430724] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7384.714664] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7384.723349] Lustre: Skipped 1 previous similar message [ 7392.044579] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 7399.317709] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7417.830189] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7417.834765] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7417.873636] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7417.883578] Lustre: Skipped 4 previous similar messages [ 7422.357992] Lustre: server umount lustre-MDT0000 complete [ 7422.948446] LustreError: 162881:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7422.964478] LustreError: 162881:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7427.197458] LustreError: 161554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445502 with bad export cookie 471804300646979918 [ 7427.219715] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7427.221854] LustreError: 161554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7428.071159] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7428.089330] Lustre: Skipped 2 previous similar messages [ 7428.099494] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7428.110156] Lustre: Skipped 1 previous similar message [ 7432.101496] LustreError: 161566:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7432.114808] LustreError: 161566:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7433.823500] Lustre: server umount lustre-MDT0001 complete [ 7442.286954] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7456.470948] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7456.879384] LustreError: 165438:0:(llog.c:1655:llog_backup()) MGC192.168.202.124@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7456.896339] Lustre: 165438:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.124@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7457.763569] LustreError: 165443:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7491.624045] LustreError: 165443:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7491.660882] LustreError: 165443:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 23 previous similar messages [ 7518.177548] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7518.188698] Lustre: 165438:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.124@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7557.161399] LustreError: 165464:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7557.176319] LustreError: 165464:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 40 previous similar messages [ 7559.179052] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7564.266418] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7566.063529] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 7577.118501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7587.821083] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 7602.672199] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7602.793509] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 7610.675478] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7612.346677] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7618.844027] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7620.917327] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7637.257495] Lustre: server umount lustre-MDT0000 complete [ 7638.497109] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7638.499462] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7638.528360] Lustre: Skipped 2 previous similar messages [ 7641.018673] LustreError: 161554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445716 with bad export cookie 471804300646986407 [ 7641.023530] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7641.030433] LustreError: 161554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7641.371122] Lustre: server umount lustre-MDT0001 complete [ 7655.724578] Lustre: server umount lustre-OST0000 complete [ 7677.809885] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_hostid [ 7687.085717] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 7726.915532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7727.149908] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7727.206304] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7727.387485] Lustre: lustre-MDT0000: new disk, initializing [ 7727.524314] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7727.531242] Lustre: Skipped 1 previous similar message [ 7727.586215] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7733.962170] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 7745.113045] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7745.266950] Lustre: 170545:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7745.564860] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7749.765188] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 7759.537812] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7765.228874] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7772.724204] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7772.945243] Lustre: lustre-OST0000: new disk, initializing [ 7772.952513] Lustre: Skipped 1 previous similar message [ 7772.955955] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7772.960243] Lustre: Skipped 2 previous similar messages [ 7772.969479] Lustre: 172517:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7778.899377] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7778.908943] Lustre: Skipped 1 previous similar message [ 7778.920793] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7779.026530] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7780.247749] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 7790.468706] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7795.474548] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7795.630962] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7799.998184] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7800.213062] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7804.907044] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7804.910678] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7804.913305] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7804.913313] Lustre: Skipped 2 previous similar messages [ 7804.927840] Lustre: Skipped 1 previous similar message [ 7808.583948] Lustre: server umount lustre-OST0000 complete [ 7818.950806] Lustre: server umount lustre-MDT0000 complete [ 7820.257402] LustreError: 171724:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7820.273658] LustreError: 171724:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 41 previous similar messages [ 7823.348616] LustreError: 170538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445898 with bad export cookie 471804300647003214 [ 7823.349188] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7823.361387] LustreError: 170538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7823.821337] Lustre: server umount lustre-MDT0001 complete [ 7833.629701] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 10:31:47 (1788445907) [ 7841.354403] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7847.151585] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 7857.394335] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7863.694411] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 7871.644704] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7877.266802] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7883.801914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7890.576610] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 7899.158678] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7914.745432] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 10:33:08 (1788445988) [ 7923.169543] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7923.175348] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7923.197326] Lustre: Skipped 2 previous similar messages [ 7923.204589] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7923.233572] Lustre: Skipped 4 previous similar messages [ 7926.419217] Lustre: server umount lustre-OST0000 complete [ 7945.184587] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7945.538459] Lustre: server umount lustre-MDT0000 complete [ 7949.166814] LustreError: 174911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446024 with bad export cookie 471804300647004096 [ 7949.177699] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7949.179188] LustreError: 174911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7949.514331] Lustre: server umount lustre-MDT0001 complete [ 7959.646082] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [ 7962.377947] Key type lgssc unregistered [ 7962.620290] LNet: 178659:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7962.630530] LNetError: 178659:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7963.688140] LNet: Removed LNI 192.168.202.124@tcp [ 7964.737560] Key type .llcrypt unregistered [ 7964.739215] Key type ._llcrypt unregistered [ 7983.881812] Key type ._llcrypt registered [ 7983.885599] Key type .llcrypt registered [ 7984.018266] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 7985.330259] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7985.462881] alg: No test for adler32 (adler32-zlib) [ 7986.525412] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 7986.829209] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [ 7988.520181] Key type lgssc registered [ 7989.445168] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7996.840877] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7996.868996] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8012.448608] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8017.568400] LustreError: 179900:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.124@tcp: failed processing log, type 4: rc = -110 [ 8022.689083] LustreError: 179900:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8022.702987] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 8022.715172] LustreError: MGC192.168.202.124@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 8022.730258] LustreError: 179900:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-OST0000: -22 [ 8022.741454] LustreError: 179900:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -22 [ 8022.759462] LustreError: 179900:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 8022.885155] Lustre: server umount lustre-OST0000 complete [ 8022.888537] LustreError: 179900:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 8034.249586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8035.965574] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8042.020703] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 8046.570106] LustreError: 180400:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8046.595424] LustreError: 180400:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8051.703841] LustreError: 180401:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8052.482097] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8053.019525] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8057.939499] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 8065.810714] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8071.662738] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8078.533360] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8078.842575] LustreError: 182217:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8078.859193] Lustre: 182217:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.202.124@tcp: invalid local config log lustre-OST0000: rc = -22 [ 8078.892717] LustreError: 182221:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8079.083166] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8084.724728] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 8091.935768] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8094.180527] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8094.190792] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8094.198757] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8096.739404] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8096.742419] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8096.758553] Lustre: Skipped 2 previous similar messages [ 8101.859240] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8101.869086] Lustre: Skipped 2 previous similar messages [ 8106.980829] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8106.993430] Lustre: Skipped 2 previous similar messages [ 8108.000278] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8108.533655] Lustre: server umount lustre-MDT0000 complete [ 8112.101146] LustreError: 181129:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8112.130570] LustreError: 181129:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8112.455123] LustreError: 180380:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446187 with bad export cookie 13859143643474308763 [ 8112.461697] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8112.472926] LustreError: 180380:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8112.686618] Lustre: server umount lustre-MDT0001 complete [ 8127.043539] Lustre: server umount lustre-OST0000 complete [ 8136.771525] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_hostid [ 8147.180395] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 8182.089497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8182.350785] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8182.380866] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8182.463565] Lustre: lustre-MDT0000: new disk, initializing [ 8182.528671] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8182.540814] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8186.777991] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 8198.051466] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8198.132501] Lustre: 185742:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8198.174933] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8198.177930] Lustre: Skipped 1 previous similar message [ 8198.251126] Lustre: lustre-MDT0001: new disk, initializing [ 8198.333257] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8198.370986] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8198.377499] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8202.833458] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 8212.241725] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8217.974365] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8225.226852] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8225.675041] Lustre: lustre-OST0000: new disk, initializing [ 8225.682964] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8225.699622] Lustre: 187713:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8225.824941] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8232.457529] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 8233.046531] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8233.068133] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8233.272773] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8241.868630] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8246.281023] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8246.564076] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8251.152977] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8251.365484] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8253.924780] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8253.929271] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8253.945560] Lustre: Skipped 1 previous similar message [ 8253.954113] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8253.978951] Lustre: Skipped 1 previous similar message [ 8259.790266] Lustre: server umount lustre-OST0000 complete [ 8265.190018] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8265.197896] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8265.216305] Lustre: Skipped 1 previous similar message [ 8265.234419] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8265.237392] Lustre: Skipped 2 previous similar messages [ 8269.284993] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8270.895610] Lustre: server umount lustre-MDT0000 complete [ 8274.401328] LustreError: 185754:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8274.433557] LustreError: 185754:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8275.096254] LustreError: 186538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446350 with bad export cookie 13859143643474309477 [ 8275.100571] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8275.102864] LustreError: 186538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8275.511849] Lustre: server umount lustre-MDT0001 complete [ 8285.838480] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 10:39:19 (1788446359) [ 8305.016840] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [ 8308.148441] Key type lgssc unregistered [ 8308.505410] LNet: 190657:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8308.518238] LNetError: 190657:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8308.529828] LNet: Removed LNI 192.168.202.124@tcp [ 8309.728537] Key type .llcrypt unregistered [ 8309.733920] Key type ._llcrypt unregistered [ 8328.636390] Key type ._llcrypt registered [ 8328.639123] Key type .llcrypt registered [ 8328.766423] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 8329.758687] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8329.924669] alg: No test for adler32 (adler32-zlib) [ 8330.941538] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 8331.154267] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [ 8332.936326] Key type lgssc registered [ 8333.869234] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8340.908581] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8340.924861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8342.306968] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8346.654670] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 8352.738901] LustreError: 191874:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8352.762602] LustreError: 191874:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8355.792739] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8356.308844] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8362.176909] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 8370.553810] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8376.902693] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8385.300501] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8385.501149] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8385.503791] LustreError: 193738:0:(llog.c:1696:llog_backup()) MGC192.168.202.124@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8385.509970] Lustre: 193738:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.202.124@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8385.623776] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8393.850676] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 8401.620254] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8406.499535] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8406.501271] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8406.516457] Lustre: Skipped 2 previous similar messages [ 8406.528666] Lustre: Skipped 2 previous similar messages [ 8410.527280] Lustre: server umount lustre-MDT0000 complete [ 8411.628158] LustreError: 193397:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8411.651428] LustreError: 193397:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8414.809832] LustreError: 191856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446489 with bad export cookie 9971362649653116871 [ 8414.818466] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8414.823398] LustreError: 191856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8415.293965] Lustre: server umount lustre-MDT0001 complete [ 8429.318620] Lustre: server umount lustre-OST0000 complete [ 8438.400113] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_hostid [ 8449.097612] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 8485.117654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8485.297088] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8485.320438] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8485.381617] Lustre: lustre-MDT0000: new disk, initializing [ 8485.447420] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8485.461546] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8490.159362] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 8500.898786] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8501.044389] Lustre: 197260:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8501.090218] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8501.091728] Lustre: Skipped 1 previous similar message [ 8501.093883] LustreError: 198045:0:(llog.c:1696:llog_backup()) MGC192.168.202.124@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8501.097773] LustreError: 198045:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8501.103549] Lustre: 198045:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.124@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8501.130993] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8501.138753] Lustre: Skipped 1 previous similar message [ 8501.297943] Lustre: lustre-MDT0001: new disk, initializing [ 8501.459937] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8501.511597] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8501.529645] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8506.061912] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 8516.042790] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8522.970658] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8531.671896] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8532.201768] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8532.211554] Lustre: Skipped 1 previous similar message [ 8532.218209] LustreError: 199208:0:(llog.c:1696:llog_backup()) MGC192.168.202.124@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8532.236044] LustreError: 199208:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8532.253841] Lustre: 199208:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.124@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8532.284993] Lustre: 199208:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8532.361490] Lustre: lustre-OST0000: new disk, initializing [ 8532.370931] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8532.386178] Lustre: 199233:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8532.537944] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8539.520910] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 8541.766478] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8541.779566] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8541.918600] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8549.172653] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8554.018575] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8554.270613] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8559.465420] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8559.750447] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8562.659927] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8562.669659] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8562.672318] LustreError: Skipped 1 previous similar message [ 8562.674601] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8562.715655] Lustre: Skipped 1 previous similar message [ 8567.751469] Lustre: server umount lustre-OST0000 complete [ 8572.903170] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8572.921313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8572.929227] Lustre: Skipped 2 previous similar messages [ 8578.025124] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8578.034463] Lustre: Skipped 1 previous similar message [ 8582.123557] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8586.208709] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8586.447032] Lustre: server umount lustre-MDT0000 complete [ 8587.234609] LustreError: 197273:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8587.269367] LustreError: 197273:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8590.511619] LustreError: 197254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446665 with bad export cookie 9971362649653117592 [ 8590.528265] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8590.546070] LustreError: 197254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8590.858378] Lustre: server umount lustre-MDT0001 complete [ 8601.257786] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 10:44:34 (1788446674) [ 8617.282213] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 8627.249354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8627.566191] LustreError: 202361:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8627.604132] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8632.059884] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 8632.803688] LustreError: 202362:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8641.255315] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8646.775653] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 8649.923524] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8656.953417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8663.681625] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 8672.465515] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8672.636705] Lustre: lustre-OST0001: new disk, initializing [ 8672.640715] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8672.647384] Lustre: 204671:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8672.790242] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8672.796416] Lustre: Skipped 2 previous similar messages [ 8677.905988] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8677.910810] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8677.977752] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8679.325372] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 8690.289409] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8697.829815] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8699.777855] Lustre: DEBUG MARKER: ADD OST3 [ 8704.550891] loop: module loaded [ 8704.609147] loop0: detected capacity change from 0 to 204800000 [ 8704.733504] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8704.761218] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8704.770634] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8710.789385] loop0: detected capacity change from 0 to 204800000 [ 8712.237828] Lustre: DEBUG MARKER: STOP OST3 [ 8717.646245] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8717.915847] Lustre: lustre-OST0003: new disk, initializing [ 8717.919889] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8717.925254] Lustre: 206655:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8719.737117] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8719.884439] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8719.896146] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8719.932905] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8721.760275] LustreError: 206838:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8722.335690] Lustre: Failing over lustre-OST0003 [ 8722.501510] Lustre: server umount lustre-OST0003 complete [ 8724.981265] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8724.983279] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8724.986965] LustreError: 203849:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8724.986976] LustreError: 203849:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8725.019911] Lustre: Skipped 2 previous similar messages [ 8725.333200] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 8733.667554] Lustre: DEBUG MARKER: START OST3 again [ 8740.312318] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8740.669108] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8740.674777] Lustre: Skipped 1 previous similar message [ 8740.704666] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8742.311918] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8742.591158] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8742.605690] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8748.882408] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 8772.119432] Lustre: Failing over lustre-OST0003 [ 8772.269371] Lustre: server umount lustre-OST0003 complete [ 8773.602124] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8773.608279] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8773.610898] LustreError: 203849:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8773.610912] LustreError: 203849:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 8773.667946] LustreError: Skipped 1 previous similar message [ 8792.042281] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8792.057915] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8792.065027] Lustre: Skipped 2 previous similar messages [ 8792.091653] Lustre: Skipped 4 previous similar messages [ 8797.935195] Lustre: server umount lustre-MDT0000 complete [ 8801.275552] LustreError: 202341:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446876 with bad export cookie 9971362649653118474 [ 8801.299028] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8801.302358] LustreError: 202341:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8806.311554] LustreError: 202356:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8806.338226] LustreError: 202356:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 8807.810650] Lustre: server umount lustre-MDT0001 complete [ 8811.903405] Lustre: server umount lustre-OST0000 complete [ 8815.247211] Lustre: server umount lustre-OST0001 complete [ 8821.203232] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_hostid [ 8829.537063] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 8865.724675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8865.929970] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8865.954556] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8866.022233] Lustre: lustre-MDT0000: new disk, initializing [ 8866.136323] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8870.543196] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 8880.647923] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8880.758261] Lustre: 212128:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8880.906991] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8880.912983] Lustre: Skipped 1 previous similar message [ 8880.946783] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8885.049881] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 8894.212181] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8899.673284] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8906.033751] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8906.329253] Lustre: lustre-OST0000: new disk, initializing [ 8906.332521] Lustre: Skipped 1 previous similar message [ 8906.340146] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8906.344152] Lustre: Skipped 2 previous similar messages [ 8906.349181] Lustre: 214095:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8907.754839] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8907.769326] Lustre: Skipped 1 previous similar message [ 8907.776401] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8907.908110] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8912.051859] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 8921.084816] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8924.595414] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8924.799903] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8929.020642] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8929.268768] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8933.346404] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8933.357059] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8933.367608] Lustre: Skipped 4 previous similar messages [ 8933.377395] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8933.387669] Lustre: Skipped 9 previous similar messages [ 8937.237588] Lustre: server umount lustre-OST0000 complete [ 8942.566208] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8942.575345] LustreError: Skipped 1 previous similar message [ 8955.364346] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8955.811519] Lustre: server umount lustre-MDT0000 complete [ 8958.951276] LustreError: 212133:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8959.009122] LustreError: 212133:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8959.897939] LustreError: 212119:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447035 with bad export cookie 9971362649653120763 [ 8959.916741] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8959.925238] LustreError: 212119:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8960.338879] Lustre: server umount lustre-MDT0001 complete [ 8964.324797] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 10:50:37 (1788447037) [ 8979.649529] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_hostid [ 8990.034988] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 9029.232735] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9029.522030] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9029.577507] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9029.698776] Lustre: lustre-MDT0000: new disk, initializing [ 9029.872031] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9035.724461] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 9047.481298] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9047.690846] Lustre: 218703:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9047.954521] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9052.885176] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 9062.167668] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9067.009858] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9073.404579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9073.690578] Lustre: 220672:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9075.747643] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9079.934745] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 9090.712698] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9094.968617] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9095.158891] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9100.161138] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9100.410929] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9106.414197] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9106.419771] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9106.422359] LustreError: Skipped 1 previous similar message [ 9106.428044] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9106.428052] Lustre: Skipped 7 previous similar messages [ 9106.452558] Lustre: Skipped 4 previous similar messages [ 9108.702265] Lustre: server umount lustre-OST0000 complete [ 9127.392470] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9127.658724] Lustre: server umount lustre-MDT0000 complete [ 9132.001753] LustreError: 218710:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9132.017286] LustreError: 218710:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9132.377498] LustreError: 218695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447207 with bad export cookie 9971362649653121652 [ 9132.397172] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9132.408819] LustreError: 218695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9133.022990] Lustre: server umount lustre-MDT0001 complete [ 9142.946153] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9143.526965] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9143.540451] Lustre: Skipped 4 previous similar messages [ 9147.639230] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 9155.115881] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9159.151623] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 9165.890578] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9171.390717] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9177.146453] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9183.677441] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 9191.888178] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9204.200366] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9204.206778] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9204.225803] Lustre: Skipped 2 previous similar messages [ 9204.234190] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9204.234203] Lustre: Skipped 8 previous similar messages [ 9204.257394] LustreError: Skipped 1 previous similar message [ 9208.189121] Lustre: server umount lustre-OST0000 complete [ 9218.439452] Lustre: server umount lustre-MDT0000 complete [ 9222.176741] LustreError: 222875:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447297 with bad export cookie 9971362649653122527 [ 9222.191701] LustreError: 222875:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9222.197693] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9222.815793] Lustre: server umount lustre-MDT0001 complete [ 9333.348433] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [ 9336.583350] Key type lgssc unregistered [ 9336.978913] LNet: 226555:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9336.991349] LNetError: 226555:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9337.024397] LNet: Removed LNI 192.168.202.124@tcp [ 9338.078239] Key type .llcrypt unregistered [ 9338.080646] Key type ._llcrypt unregistered [ 9367.677625] Key type ._llcrypt registered [ 9367.679636] Key type .llcrypt registered [ 9367.774628] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_hostid [ 9385.071816] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 9386.739265] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9386.802403] alg: No test for adler32 (adler32-zlib) [ 9387.966805] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 9388.360693] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [ 9390.112176] Key type lgssc registered [ 9391.512145] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9428.186122] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9428.207872] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9429.538216] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9429.574410] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9429.727287] Lustre: lustre-MDT0000: new disk, initializing [ 9429.858303] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9429.872341] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9434.199711] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 9444.527208] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9444.691806] Lustre: 229906:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9444.748368] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9444.752513] Lustre: Skipped 1 previous similar message [ 9444.856916] Lustre: lustre-MDT0001: new disk, initializing [ 9444.977077] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9445.020386] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9445.037133] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9450.329702] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 9460.128922] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9467.473497] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9475.529432] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9475.767155] Lustre: lustre-OST0000: new disk, initializing [ 9475.770641] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9475.776424] Lustre: 231873:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9475.864855] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9483.484720] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9483.521179] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9483.838925] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9484.948772] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 9494.333834] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9499.564929] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9499.730360] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9504.883507] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9505.083298] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9509.345541] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9509.346993] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9509.353743] LustreError: Skipped 1 previous similar message [ 9509.359412] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9509.368362] Lustre: Skipped 1 previous similar message [ 9509.380873] Lustre: Skipped 1 previous similar message [ 9514.473428] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9514.478862] Lustre: Skipped 1 previous similar message [ 9519.602096] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9519.617339] Lustre: Skipped 1 previous similar message [ 9521.120334] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9521.386810] Lustre: server umount lustre-OST0000 complete [ 9526.758379] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9526.770323] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9526.797618] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9529.830068] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9532.404875] Lustre: server umount lustre-MDT0000 complete [ 9537.612274] LustreError: 229899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447612 with bad export cookie 5237989045927923130 [ 9537.635395] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9538.477430] Lustre: server umount lustre-MDT0001 complete [ 9561.879851] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 9574.218515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9574.842845] LustreError: 234811:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9574.873060] LustreError: 234811:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9574.928543] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9580.007084] LustreError: 234812:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9580.196312] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 9585.127422] LustreError: 234812:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9590.414548] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9590.734797] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9597.120256] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 9600.847877] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9609.714261] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9610.014561] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9617.554631] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 9627.481705] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9627.643448] Lustre: lustre-OST0001: new disk, initializing [ 9627.646657] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9627.654495] Lustre: 237130:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9627.743623] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9633.958516] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 9635.458538] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9635.472547] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9635.557109] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9644.934499] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9652.357729] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9665.124824] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 11:02:18 (1788447738) [ 9667.555338] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9667.565434] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9667.578443] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9667.586773] Lustre: Skipped 2 previous similar messages [ 9671.656850] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9671.681340] Lustre: Skipped 1 previous similar message [ 9673.439559] Lustre: server umount lustre-MDT0000 complete [ 9676.775376] LustreError: 234806:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9676.791943] LustreError: 234806:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 9677.170340] LustreError: 234791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447752 with bad export cookie 5237989045927924012 [ 9677.175262] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9677.181971] LustreError: 234791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 9677.590703] Lustre: server umount lustre-MDT0001 complete [ 9691.265994] Lustre: server umount lustre-OST0000 complete [ 9705.638837] Lustre: server umount lustre-OST0001 complete [ 9714.347747] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_hostid [ 9725.165953] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 9764.934384] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9765.498423] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9765.670855] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9765.927232] Lustre: lustre-MDT0000: new disk, initializing [ 9766.039608] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9766.073242] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9771.811176] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 9786.714669] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9786.870843] Lustre: 241076:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9786.961984] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9786.969412] Lustre: Skipped 1 previous similar message [ 9787.134273] Lustre: lustre-MDT0001: new disk, initializing [ 9787.429833] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9787.451164] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9793.558557] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 9804.542745] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9812.203692] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9820.346862] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9820.611906] Lustre: lustre-OST0000: new disk, initializing [ 9820.615819] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9820.625120] Lustre: 243044:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9822.700742] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9822.719844] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9822.882974] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9829.140827] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 9840.752355] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9846.889080] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9847.057394] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9851.227882] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9851.654854] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9853.420331] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9853.433762] Lustre: Skipped 1 previous similar message [ 9853.437082] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9853.437140] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9853.443108] Lustre: Skipped 5 previous similar messages [ 9859.909079] Lustre: server umount lustre-OST0000 complete [ 9871.488121] Lustre: server umount lustre-MDT0000 complete [ 9873.918727] LustreError: 241084:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9873.952144] LustreError: 241084:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9876.095092] LustreError: 241069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447951 with bad export cookie 5237989045927926154 [ 9876.100725] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9876.113385] LustreError: 241069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9876.571423] Lustre: server umount lustre-MDT0001 complete [ 9886.174040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9886.727163] LustreError: 245265:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9886.839460] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9886.848905] Lustre: Skipped 2 previous similar messages [ 9893.165475] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 9903.233457] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9909.321503] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 9918.010407] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9925.110530] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9933.521216] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9940.576947] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 9948.631697] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10097.121316] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10097.137223] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10097.168267] Lustre: Skipped 3 previous similar messages [10097.191718] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10097.204040] Lustre: Skipped 4 previous similar messages [10102.038043] Lustre: server umount lustre-OST0000 complete [10108.898161] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10112.338609] Lustre: server umount lustre-MDT0000 complete [10115.554084] LustreError: 245266:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10115.581184] LustreError: 245266:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [10116.082868] LustreError: 246005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448191 with bad export cookie 5237989045927927036 [10116.087324] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10116.090975] LustreError: 246005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10116.488679] Lustre: server umount lustre-MDT0001 complete [10217.169781] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [10220.243365] Key type lgssc unregistered [10220.603915] LNet: 248871:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10220.616799] LNetError: 248871:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10220.654742] LNet: Removed LNI 192.168.202.124@tcp [10221.903142] Key type .llcrypt unregistered [10221.910366] Key type ._llcrypt unregistered [10252.251688] Key type ._llcrypt registered [10252.253421] Key type .llcrypt registered [10252.370301] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_hostid [10269.183586] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [10270.378697] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10270.396582] alg: No test for adler32 (adler32-zlib) [10271.480305] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [10271.777631] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [10273.504201] Key type lgssc registered [10274.640502] Lustre: Echo OBD driver; http://www.lustre.org/ [10323.853660] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [10334.061888] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10334.124436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10335.445393] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10335.498591] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10335.605975] Lustre: lustre-MDT0000: new disk, initializing [10335.720430] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10335.744458] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10340.590571] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [10351.565429] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10351.692084] Lustre: 252942:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10351.794796] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10351.801278] Lustre: Skipped 1 previous similar message [10351.931309] Lustre: lustre-MDT0001: new disk, initializing [10352.007726] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10352.041740] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10352.057543] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10356.387664] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [10362.671193] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10371.127529] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10371.664460] Lustre: lustre-OST0000: new disk, initializing [10371.677248] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10371.683835] Lustre: 254575:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10371.898474] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10372.980303] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10372.996831] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10373.248127] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10381.686336] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [10395.225791] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10395.563795] Lustre: lustre-OST0001: new disk, initializing [10395.572629] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10395.582512] Lustre: 255449:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10395.825744] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10401.328552] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10401.336349] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10401.404699] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10404.315619] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [10415.245518] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10420.658356] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10430.000205] Lustre: server umount lustre-MDT0000 complete [10431.975590] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10431.998176] Lustre: Skipped 3 previous similar messages [10433.528507] LustreError: 252933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448508 with bad export cookie 18065789118976109645 [10433.539044] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10433.547295] LustreError: 252933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10436.208607] Lustre: server umount lustre-MDT0001 complete [10450.875601] Lustre: server umount lustre-OST0000 complete [10452.961283] Lustre: 250448:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788448512/real 1788448512] req@ffff9e83b7be0000 x1875324012934016/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788448528 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10453.005536] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10454.309066] Lustre: 250449:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788448513/real 1788448513] req@ffff9e83bb0a0380 x1875324012934272/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788448529 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10455.153487] Lustre: server umount lustre-OST0001 complete [10468.966455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10478.902926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10486.745866] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10497.078735] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10518.795270] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [10530.477271] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10530.499244] Lustre: lustre-MDT0000: reset Object Index mappings [10531.078134] LustreError: 260454:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10531.105647] LustreError: 260454:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10531.161095] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10535.886089] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [10536.422867] LustreError: 260455:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10541.537418] LustreError: 260454:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10544.890632] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10544.906230] Lustre: lustre-MDT0001: reset Object Index mappings [10545.264985] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10549.414962] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [10552.291703] Lustre: 261595:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10558.963131] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10568.509530] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [10571.826503] LustreError: 261951:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10575.918069] LustreError: 261951:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10575.946171] LustreError: 261951:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10575.956311] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10581.038912] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10581.680095] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10581.689403] Lustre: Skipped 1 previous similar message [10587.179794] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10588.922438] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [10596.751940] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10600.394250] Lustre: 263470:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10606.562253] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10606.567247] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10606.583188] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10607.529205] Lustre: server umount lustre-MDT0000 complete [10607.593534] LustreError: 260455:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10607.618330] LustreError: 260455:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10611.352185] LustreError: 260434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448686 with bad export cookie 18065789118976116435 [10611.356776] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10611.369460] LustreError: 260434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10611.796501] Lustre: server umount lustre-MDT0001 complete [10626.459298] Lustre: server umount lustre-OST0000 complete [10628.066573] Lustre: 250449:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788448687/real 1788448687] req@ffff9e839d5c8000 x1875324012983040/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788448703 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10628.107395] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10628.128422] Lustre: Skipped 3 previous similar messages [10630.640937] Lustre: server umount lustre-OST0001 complete [10649.420202] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 11:18:43 (1788448723) [10666.039826] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_hostid [10676.376140] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [10712.018661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10712.125949] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10712.138038] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10712.191174] Lustre: lustre-MDT0000: new disk, initializing [10712.250463] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10712.272046] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10716.086536] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [10727.524271] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10727.710386] Lustre: 267727:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10727.805346] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10727.813132] Lustre: Skipped 1 previous similar message [10727.923309] Lustre: lustre-MDT0001: new disk, initializing [10728.124111] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10728.143301] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10733.036682] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [10742.541229] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10748.563816] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10756.655833] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10756.954885] Lustre: lustre-OST0000: new disk, initializing [10756.960777] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10756.970315] Lustre: 269696:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10758.827665] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10758.836897] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10759.039581] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10764.672790] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [10773.681933] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10778.697881] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10778.852441] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10783.282828] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10783.569401] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10789.872921] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10789.877382] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10789.886685] LustreError: Skipped 1 previous similar message [10789.889177] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10789.918218] Lustre: Skipped 1 previous similar message [10791.956267] Lustre: server umount lustre-OST0000 complete [10799.585669] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10799.596412] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10799.612544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10799.626528] Lustre: Skipped 1 previous similar message [10802.885811] Lustre: server umount lustre-MDT0000 complete [10805.224753] LustreError: 267738:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10805.256540] LustreError: 267738:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [10806.959031] LustreError: 267719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448882 with bad export cookie 18065789118976118528 [10806.962050] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10806.976166] LustreError: 267719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10807.423276] Lustre: server umount lustre-MDT0001 complete [10828.033402] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [10837.715236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10838.039887] LustreError: 272651:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10838.085912] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10838.091892] Lustre: Skipped 2 previous similar messages [10842.578649] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [10850.728896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10856.771503] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [10859.975468] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10866.366809] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10872.841804] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [10882.853765] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10883.055793] Lustre: lustre-OST0001: new disk, initializing [10883.060917] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10883.067205] Lustre: 274960:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10884.284208] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10884.294656] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10884.453144] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10889.118889] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [10897.790545] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10900.991080] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10905.443654] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10905.513452] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10915.309565] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10915.318876] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10915.346836] Lustre: Skipped 3 previous similar messages [10915.365729] Lustre: Skipped 5 previous similar messages [10919.503342] Lustre: server umount lustre-MDT0000 complete [10920.421778] LustreError: 273379:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10920.433188] LustreError: 273379:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10922.724370] LustreError: 274958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448997 with bad export cookie 18065789118976119410 [10922.729953] LustreError: 274958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10922.733379] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10923.004765] Lustre: server umount lustre-MDT0001 complete [10936.439259] Lustre: server umount lustre-OST0000 complete [10950.779064] Lustre: server umount lustre-OST0001 complete [10967.202750] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [10976.936721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10977.522202] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10977.531284] Lustre: Skipped 3 previous similar messages [10982.132660] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [10989.589775] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10994.950813] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [10997.786647] Lustre: 279449:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [11003.892558] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11010.962669] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [11020.661249] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11022.079346] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [11022.079522] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [11027.365955] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [11036.127964] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11040.978653] Lustre: 281327:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11047.354421] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [11047.433939] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [11063.578979] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 11:25:36 (1788449136) [11082.212374] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11082.225327] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11082.231442] Lustre: Skipped 1 previous similar message [11082.235615] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11086.378123] Lustre: server umount lustre-MDT0000 complete [11088.868648] LustreError: 278309:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11088.885604] LustreError: 278309:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [11091.047888] LustreError: 282526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449166 with bad export cookie 18065789118976126011 [11091.059754] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11091.067297] LustreError: 282526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11094.003394] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11094.010260] Lustre: Skipped 4 previous similar messages [11097.454253] Lustre: server umount lustre-MDT0001 complete [11107.079036] Lustre: server umount lustre-OST0000 complete [11110.657817] Lustre: server umount lustre-OST0001 complete [11118.220406] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_hostid [11129.224609] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [11179.808383] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [11192.379256] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11192.613737] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11192.641299] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11192.716561] Lustre: lustre-MDT0000: new disk, initializing [11192.807193] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11197.484615] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [11208.315476] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11208.409801] Lustre: 285868:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11208.632468] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11213.340705] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [11226.680078] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11227.188930] Lustre: 287506:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11228.903547] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11234.647554] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [11246.714545] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11246.966160] Lustre: 288377:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11247.072136] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11247.075108] Lustre: Skipped 6 previous similar messages [11248.568309] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11248.582722] Lustre: Skipped 1 previous similar message [11248.689947] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11252.306707] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [11262.351934] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11266.048470] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11266.052333] Lustre: Skipped 1 previous similar message [11283.050308] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 11:29:16 (1788449356) [11285.473452] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11285.478348] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11285.512340] Lustre: Skipped 5 previous similar messages [11285.530455] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11285.546109] Lustre: Skipped 2 previous similar messages [11287.811571] Lustre: server umount lustre-MDT0000 complete [11291.527810] LustreError: 285862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449366 with bad export cookie 18065789118976133732 [11291.529301] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11291.541138] LustreError: 285862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11291.944708] Lustre: server umount lustre-MDT0001 complete [11305.215347] Lustre: server umount lustre-OST0000 complete [11318.196863] Lustre: server umount lustre-OST0001 complete [11327.648379] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11333.736062] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [11344.296922] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11350.893194] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [11359.983661] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11366.433777] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11374.231248] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11382.015297] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [11389.098671] LustreError: 292456:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11389.125880] LustreError: 292456:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [11389.290115] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11417.938151] loop1: detected capacity change from 0 to 204800000 [11418.185732] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11418.199882] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11418.232134] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11423.395540] loop1: detected capacity change from 0 to 204800000 [11430.965959] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11431.157400] LustreError: 290614:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs4646: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11431.165950] LustreError: 290614:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs4646 [11431.168235] LustreError: 290614:0:(mgs_handler.c:517:mgs_target_reg()) fs4646-MDT0001: Error checking target index: rc = -13 [11431.173771] LustreError: fs4646-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11431.181397] LustreError: 294367:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -13 [11431.194147] LustreError: 294367:0:(tgt_mount.c:2132:server_put_super()) no obd fs4646-MDT0001 [11431.207157] LustreError: 294367:0:(tgt_mount.c:116:server_deregister_mount()) fs4646-MDT0001 not registered [11431.242695] Lustre: server umount fs4646-MDT0001 complete [11431.249435] LustreError: 294367:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11440.726294] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11440.796637] Lustre: Setting parameter fs4646-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs4646-MDT0001 [11440.838537] Lustre: srv-fs4646-MDT0001: No data found on store. Initialize space. [11440.842356] Lustre: Skipped 4 previous similar messages [11440.921539] Lustre: fs4646-MDT0001: new disk, initializing [11440.927536] Lustre: Skipped 3 previous similar messages [11441.055253] Lustre: srv-fs4646-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11441.065984] Lustre: 294833:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs4646-MDT0001-mdtlov: can't create new llog: rc = -115 [11441.077981] LustreError: 294833:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs4646-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11445.846491] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [11452.469065] Lustre: server umount fs4646-MDT0001 complete [11466.895335] loop1: detected capacity change from 0 to 204800000 [11470.323985] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11474.679230] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11475.355216] Lustre: 296400:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs4646-MDT0001-mdtlov: can't create new llog: rc = -115 [11475.366079] LustreError: 296400:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs4646-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11481.298153] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [11488.016726] Lustre: server umount fs4646-MDT0001 complete [11501.030408] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11501.038300] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11501.055201] Lustre: Skipped 4 previous similar messages [11501.062274] Lustre: Skipped 1 previous similar message [11506.237287] Lustre: server umount lustre-MDT0000 complete [11515.535227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11515.786198] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11521.295408] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [11521.543856] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11529.107580] loop1: detected capacity change from 0 to 204800000 [11535.129723] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11535.743818] Lustre: 298850:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs4646-MDT0001-mdtlov: can't create new llog: rc = -115 [11535.762748] LustreError: 298850:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs4646-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11540.597161] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [11545.106536] Lustre: server umount fs4646-MDT0001 complete [11572.720906] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11572.724436] LustreError: Skipped 1 previous similar message [11572.728809] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11572.745478] Lustre: Skipped 2 previous similar messages [11576.275743] Lustre: server umount lustre-OST0000 complete [11594.209095] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11594.555867] Lustre: server umount lustre-MDT0000 complete [11598.599552] LustreError: 290606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449673 with bad export cookie 18065789118976136413 [11598.607784] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11598.615208] LustreError: 290606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11599.181523] Lustre: server umount lustre-MDT0001 complete [11610.528516] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [11614.319314] Key type lgssc unregistered [11614.640829] LNet: 301193:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11614.648625] LNetError: 301193:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11614.693385] LNet: Removed LNI 192.168.202.124@tcp [11616.147394] Key type .llcrypt unregistered [11616.151074] Key type ._llcrypt unregistered [11637.843883] Key type ._llcrypt registered [11637.846524] Key type .llcrypt registered [11638.003416] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [11639.387289] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11639.442154] alg: No test for adler32 (adler32-zlib) [11640.523775] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [11640.826467] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [11642.472555] Key type lgssc registered [11644.057231] Lustre: Echo OBD driver; http://www.lustre.org/ [11652.370892] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11652.406019] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11653.945715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11659.133356] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [11663.332225] LustreError: 302478:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11663.348253] LustreError: 302478:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11664.361864] LustreError: 302477:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11667.541092] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11667.995474] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11672.053160] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [11680.437285] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11685.726728] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11691.829280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11692.338095] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11693.375021] LustreError: 304306:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11698.549524] LustreError: 304306:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11699.153778] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [11704.805624] LustreError: 304307:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11704.826088] LustreError: 304307:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11706.456979] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11714.606322] LustreError: 304306:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11714.630232] LustreError: 304306:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11725.102286] loop1: detected capacity change from 0 to 204800000 [11725.293940] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11725.307425] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11725.318743] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11730.587747] loop1: detected capacity change from 0 to 204800000 [11735.089749] LustreError: 304305:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11735.108171] LustreError: 304305:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [11735.273483] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11735.435343] Lustre: fs19663-OST0000: new disk, initializing [11735.438713] Lustre: srv-fs19663-OST0000: No data found on store. Initialize space. [11735.445208] Lustre: 306209:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11735.535297] Lustre: fs19663-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11741.704858] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [11750.424397] Lustre: server umount fs19663-OST0000 complete [11762.943494] loop1: detected capacity change from 0 to 204800000 [11768.335100] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11768.514083] Lustre: fs19663-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11770.938845] LustreError: 304306:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11770.967278] LustreError: 304306:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 20 previous similar messages [11774.594288] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [11776.497843] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11781.607452] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11783.741780] loop2: detected capacity change from 0 to 204800000 [11784.101417] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11784.133595] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11784.182178] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11790.159482] loop2: detected capacity change from 0 to 204800000 [11798.649515] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11798.744840] LustreError: fs19663-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11798.753036] LustreError: fs19663-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11798.756681] LustreError: 309248:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -13 [11798.760556] LustreError: 309248:0:(tgt_mount.c:2132:server_put_super()) no obd fs19663-OST0001 [11798.764383] LustreError: 309248:0:(tgt_mount.c:116:server_deregister_mount()) fs19663-OST0001 not registered [11798.787141] Lustre: server umount fs19663-OST0001 complete [11798.789282] LustreError: 309248:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11810.861793] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11811.098521] Lustre: fs19663-OST0001: new disk, initializing [11811.101974] Lustre: srv-fs19663-OST0001: No data found on store. Initialize space. [11811.109989] Lustre: 309801:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11811.212638] Lustre: fs19663-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11819.182557] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [11828.606363] Lustre: server umount fs19663-OST0000 complete [11837.484534] LustreError: 304306:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11837.498905] LustreError: 304306:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 38 previous similar messages [11838.301421] Lustre: server umount fs19663-OST0001 complete [11850.223386] loop1: detected capacity change from 0 to 204800000 [11856.033881] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11856.369947] Lustre: fs19663-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11864.037394] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [11871.398279] loop2: detected capacity change from 0 to 204800000 [11877.543252] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11884.081170] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [11893.374853] Lustre: server umount fs19663-OST0000 complete [11901.949793] Lustre: server umount fs19663-OST0001 complete [11914.724659] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11914.734850] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11914.744871] Lustre: Skipped 1 previous similar message [11919.847895] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11919.861407] Lustre: Skipped 2 previous similar messages [11920.197368] Lustre: server umount lustre-OST0000 complete [11924.962386] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11924.975361] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11930.083519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11930.087518] Lustre: Skipped 2 previous similar messages [11930.850759] Lustre: server umount lustre-MDT0000 complete [11934.802634] LustreError: 302459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450009 with bad export cookie 8235143329227592900 [11934.812841] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11934.817599] LustreError: 302459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11935.134414] Lustre: server umount lustre-MDT0001 complete [11946.436398] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [11949.038961] Key type lgssc unregistered [11949.373499] LNet: 315725:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11949.388199] LNetError: 315725:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11949.409648] LNet: Removed LNI 192.168.202.124@tcp [11950.129347] Key type .llcrypt unregistered [11950.131475] Key type ._llcrypt unregistered [11979.719852] Key type ._llcrypt registered [11979.722335] Key type .llcrypt registered [11979.795076] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_hostid [11994.297502] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [11995.615901] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11995.636413] alg: No test for adler32 (adler32-zlib) [11996.701359] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [11996.943343] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [11998.592174] Key type lgssc registered [11999.831828] Lustre: Echo OBD driver; http://www.lustre.org/ [12032.295149] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12032.311427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12033.515866] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12033.543511] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12033.603840] Lustre: lustre-MDT0000: new disk, initializing [12033.662271] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12033.684202] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12038.438580] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [12048.792643] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12048.938778] Lustre: 319581:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12048.985980] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12048.989530] Lustre: Skipped 1 previous similar message [12049.078781] Lustre: lustre-MDT0001: new disk, initializing [12049.176961] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12049.215736] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12049.241852] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12053.123356] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [12062.027525] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12068.799401] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12076.825329] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12077.169572] Lustre: lustre-OST0000: new disk, initializing [12077.173295] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12077.180883] Lustre: 321548:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12077.275723] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12083.292087] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [12086.311491] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12086.326327] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12086.454334] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12092.014736] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12094.494761] Lustre: Mounted lustre-client - version 2.17.58_2_gee80f9b [12095.547190] Lustre: Unmounted lustre-client [12107.237679] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12107.254847] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12107.283247] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12107.295309] Lustre: Skipped 1 previous similar message [12109.798770] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12109.809479] Lustre: Skipped 1 previous similar message [12113.416847] Lustre: server umount lustre-OST0000 complete [12120.033652] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12120.058889] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12120.062132] Lustre: Skipped 2 previous similar messages [12120.078201] Lustre: Skipped 1 previous similar message [12123.722588] Lustre: server umount lustre-MDT0000 complete [12127.755628] LustreError: 321558:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450202 with bad export cookie 1578648510882163980 [12127.756980] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12127.772213] LustreError: 321558:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12128.213819] Lustre: server umount lustre-MDT0001 complete [12137.789979] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [12140.201261] Key type lgssc unregistered [12152.864839] LNet: 323711:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12152.869480] LNetError: 323711:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12223.527210] LNet: Removed LNI 192.168.202.124@tcp [12224.597966] Key type .llcrypt unregistered [12224.599801] Key type ._llcrypt unregistered [12241.517303] Key type ._llcrypt registered [12241.519994] Key type .llcrypt registered [12241.599619] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_hostid [12256.400845] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [12257.675932] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12257.922814] alg: No test for adler32 (adler32-zlib) [12259.078292] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [12259.357806] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [12260.968648] Key type lgssc registered [12262.071171] Lustre: Echo OBD driver; http://www.lustre.org/ [12293.660403] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 11:46:07 (1788450367) [12299.224500] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12299.248666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12300.488877] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12300.514801] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12300.573895] Lustre: lustre-MDT0000: new disk, initializing [12300.629656] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12300.641255] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12304.322129] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [12313.828362] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12313.980289] Lustre: 326933:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12314.142647] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12314.149568] Lustre: Skipped 1 previous similar message [12314.262562] Lustre: lustre-MDT0001: new disk, initializing [12314.385180] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12314.435666] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12314.446984] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12318.528681] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [12326.394788] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12330.566372] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12335.805717] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12336.013905] Lustre: lustre-OST0000: new disk, initializing [12336.017839] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12336.023260] Lustre: 328899:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12336.109389] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12342.145383] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [12344.424420] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12344.458283] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12344.600130] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12350.677528] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12365.286757] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12365.292741] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12365.304755] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12365.328783] Lustre: Skipped 1 previous similar message [12370.468481] Lustre: server umount lustre-OST0000 complete [12375.524189] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12375.526939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12375.549984] Lustre: Skipped 2 previous similar messages [12375.555791] Lustre: Skipped 1 previous similar message [12380.644127] LustreError: 329621:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12380.675650] LustreError: 329621:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12381.084245] Lustre: server umount lustre-MDT0000 complete [12385.775730] LustreError: 326938:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12385.830248] LustreError: 326938:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12385.889414] LustreError: 327727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450461 with bad export cookie 11845967881293236659 [12385.892669] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12385.915323] LustreError: 327727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12386.453815] Lustre: server umount lustre-MDT0001 complete [12402.989503] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [12406.415908] Key type lgssc unregistered [12406.874798] LNet: 330807:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12406.891309] LNetError: 330807:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12406.922274] LNet: Removed LNI 192.168.202.124@tcp [12408.083159] Key type .llcrypt unregistered [12408.085172] Key type ._llcrypt unregistered [12433.176734] Key type ._llcrypt registered [12433.180710] Key type .llcrypt registered [12433.274314] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [12434.324481] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12434.376990] alg: No test for adler32 (adler32-zlib) [12435.527282] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [12435.902985] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [12437.664173] Key type lgssc registered [12439.006458] Lustre: Echo OBD driver; http://www.lustre.org/ [12447.854506] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12447.877263] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12449.536746] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12454.146924] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [12458.982210] LustreError: 332080:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12458.995382] LustreError: 332080:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12462.375888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12462.631861] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12466.277445] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [12473.371356] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12479.117736] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12486.212953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12486.687660] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12493.486461] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [12501.770799] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12513.649969] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12515.545512] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12520.016288] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.24@tcp (stopping) [12520.421107] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12520.452889] Lustre: Skipped 1 previous similar message [12525.099559] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.24@tcp (stopping) [12525.102177] Lustre: Skipped 3 previous similar messages [12525.571874] LustreError: 332080:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12525.699869] Lustre: server umount lustre-MDT0000 complete [12529.846210] LustreError: 332062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450604 with bad export cookie 9195483206619858180 [12529.860305] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12529.861643] LustreError: 332062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12530.218717] LustreError: 332075:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12530.265686] LustreError: 332075:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12530.280090] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.24@tcp (stopping) [12530.659625] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12530.678391] Lustre: Skipped 1 previous similar message [12534.697684] LustreError: 334571:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12534.698404] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12534.718592] LustreError: 334571:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12534.756575] Lustre: Skipped 1 previous similar message [12536.395996] Lustre: server umount lustre-MDT0001 complete [12541.112711] Lustre: server umount lustre-OST0000 complete [12552.708798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12558.732542] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [12561.879338] Lustre: 336262:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.124@tcp [12564.271026] Lustre: server umount MGS complete [12574.059751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12574.550051] LustreError: 336834:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12574.570083] LustreError: 336834:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12574.629125] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12581.452768] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [12583.911856] LustreError: 336834:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12583.928500] LustreError: 336834:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12591.214505] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12591.522720] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12596.275873] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [12606.780681] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12607.283554] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12614.888236] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [12624.289466] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12625.589437] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12631.876576] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12633.541768] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12639.723507] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [12639.738961] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12639.761134] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12639.778771] Lustre: Skipped 1 previous similar message [12640.494653] Lustre: server umount lustre-OST0000 complete [12645.350246] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12645.369431] Lustre: Skipped 2 previous similar messages [12650.469672] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12650.484505] Lustre: Skipped 5 previous similar messages [12650.837461] Lustre: server umount lustre-MDT0000 complete [12654.862632] LustreError: 336815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450729 with bad export cookie 9195483206619859020 [12654.874212] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12654.887381] LustreError: 336815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12655.704099] Lustre: server umount lustre-MDT0001 complete [12668.524078] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [12671.602844] Key type lgssc unregistered [12671.960911] LNet: 340271:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12671.966920] LNetError: 340271:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12671.978934] LNet: Removed LNI 192.168.202.124@tcp [12672.905404] Key type .llcrypt unregistered [12672.907965] Key type ._llcrypt unregistered [12693.367743] Key type ._llcrypt registered [12693.370735] Key type .llcrypt registered [12693.475420] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_hostid [12709.931813] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [12711.315521] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12711.444078] alg: No test for adler32 (adler32-zlib) [12712.545782] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [12712.754368] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [12714.400179] Key type lgssc registered [12715.451819] Lustre: Echo OBD driver; http://www.lustre.org/ [12755.731589] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12755.794966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12757.079967] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12757.146656] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12757.248337] Lustre: lustre-MDT0000: new disk, initializing [12757.373169] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12757.389127] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12761.294497] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [12771.322759] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12771.435208] Lustre: 343431:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12771.478232] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12771.480993] Lustre: Skipped 1 previous similar message [12771.532258] Lustre: lustre-MDT0001: new disk, initializing [12771.615912] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12771.639504] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12771.651933] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12775.783245] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [12784.588267] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12790.204867] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12796.557697] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12796.823171] Lustre: lustre-OST0000: new disk, initializing [12796.827602] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12796.836813] Lustre: 345394:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12796.933132] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12802.634063] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [12804.141804] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12804.147932] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12804.265767] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12811.991894] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12816.472379] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12816.645496] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12821.426221] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12821.563805] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12825.062044] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12825.081467] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12825.091387] LustreError: Skipped 1 previous similar message [12825.092150] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12825.143790] Lustre: Skipped 1 previous similar message [12829.962168] Lustre: server umount lustre-OST0000 complete [12835.306925] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12835.327494] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12835.337188] Lustre: Skipped 2 previous similar messages [12839.837620] Lustre: server umount lustre-MDT0000 complete [12843.709511] LustreError: 343423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450918 with bad export cookie 11159471612159790612 [12843.743549] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12844.014962] Lustre: server umount lustre-MDT0001 complete [12852.805094] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 11:55:26 (1788450926) [12867.858347] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_hostid [12878.147246] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [12916.989430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12917.345308] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12917.386787] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12917.504933] Lustre: lustre-MDT0000: new disk, initializing [12917.631453] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12917.668240] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12922.567350] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [12935.096154] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12935.218226] Lustre: 350150:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12935.310421] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12935.314429] Lustre: Skipped 1 previous similar message [12935.379915] Lustre: lustre-MDT0001: new disk, initializing [12935.464993] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12935.500443] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12935.511139] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12941.436564] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [12952.237370] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12958.512682] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12966.160752] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12966.644373] Lustre: lustre-OST0000: new disk, initializing [12966.649672] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12966.661483] Lustre: 352120:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12966.751150] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12972.616213] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12972.630600] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12972.825902] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12973.979088] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [12985.781235] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13024.226379] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13024.228304] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13024.246681] LustreError: Skipped 1 previous similar message [13024.247529] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13024.280757] Lustre: Skipped 2 previous similar messages [13028.544550] Lustre: server umount lustre-OST0000 complete [13034.467540] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13034.474616] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13034.479831] Lustre: Skipped 1 previous similar message [13034.523559] Lustre: Skipped 2 previous similar messages [13039.294630] Lustre: server umount lustre-MDT0000 complete [13039.586324] LustreError: 350163:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13039.612087] LustreError: 350163:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [13043.291525] LustreError: 350144:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451118 with bad export cookie 11159471612159791494 [13043.294053] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13043.300032] LustreError: 350144:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [13043.687619] Lustre: server umount lustre-MDT0001 complete [13056.343785] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [13060.148512] Key type lgssc unregistered [13060.485352] LNet: 354294:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13060.499025] LNetError: 354294:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13060.519118] LNet: Removed LNI 192.168.202.124@tcp [13061.486905] Key type .llcrypt unregistered [13061.489531] Key type ._llcrypt unregistered [13081.558245] Key type ._llcrypt registered [13081.560566] Key type .llcrypt registered [13081.653911] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [13082.799132] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13082.906963] alg: No test for adler32 (adler32-zlib) [13083.865566] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [13084.074420] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [13085.720181] Key type lgssc registered [13087.034552] Lustre: Echo OBD driver; http://www.lustre.org/ [13094.463091] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13094.492269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13095.968071] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13100.368785] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [13106.152764] LustreError: 355571:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13106.163925] LustreError: 355571:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13107.574841] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13108.063966] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13113.611567] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [13122.918744] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13129.742693] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13135.852461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13136.159532] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13142.142311] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [13149.132906] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13164.005923] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13164.012223] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13164.028692] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13169.131038] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13169.140915] Lustre: Skipped 1 previous similar message [13169.396645] Lustre: server umount lustre-OST0000 complete [13174.243732] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13174.255827] Lustre: Skipped 1 previous similar message [13174.263236] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13174.271044] Lustre: Skipped 2 previous similar messages [13179.372570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13179.598662] Lustre: server umount lustre-MDT0000 complete [13183.993737] LustreError: 355554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451259 with bad export cookie 12869874871601460212 [13184.008854] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13184.393745] Lustre: server umount lustre-MDT0001 complete [13197.068928] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [13201.758080] Key type lgssc unregistered [13202.324897] LNet: 361151:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13202.338403] LNetError: 361151:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13203.375791] LNet: Removed LNI 192.168.202.124@tcp [13204.823400] Key type .llcrypt unregistered [13204.830872] Key type ._llcrypt unregistered [13226.495398] Key type ._llcrypt registered [13226.504861] Key type .llcrypt registered [13226.665511] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [13227.887605] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13227.944361] alg: No test for adler32 (adler32-zlib) [13229.317246] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [13229.762598] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [13231.556754] Key type lgssc registered [13233.137795] Lustre: Echo OBD driver; http://www.lustre.org/ [13243.592512] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13243.622481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13245.172217] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13250.427400] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [13255.650421] LustreError: 364969:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13255.686693] LustreError: 364969:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13259.973587] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13260.613736] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13265.890354] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [13274.660867] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13280.587300] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13288.561137] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13289.050286] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13296.694617] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [13307.278354] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13320.686951] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13320.695630] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13320.710834] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13321.197052] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13323.825473] Lustre: server umount lustre-OST0000 complete [13331.425925] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13331.442756] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13331.445597] Lustre: Skipped 2 previous similar messages [13331.463411] Lustre: Skipped 3 previous similar messages [13336.546337] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13336.562095] Lustre: Skipped 1 previous similar message [13341.666635] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13341.677329] Lustre: Skipped 1 previous similar message [13343.200138] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13343.808807] Lustre: server umount lustre-MDT0000 complete [13347.810539] LustreError: 366739:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13347.825963] LustreError: 366739:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13348.039916] LustreError: 364891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451423 with bad export cookie 1311092848541765686 [13348.041900] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13348.063696] LustreError: 364891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13348.824952] Lustre: server umount lustre-MDT0001 complete [13361.343852] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [13364.918291] Key type lgssc unregistered [13365.288396] LNet: 372715:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13365.299925] LNetError: 372715:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13365.342521] LNet: Removed LNI 192.168.202.124@tcp [13366.482870] Key type .llcrypt unregistered [13366.492842] Key type ._llcrypt unregistered [13386.065938] Key type ._llcrypt registered [13386.068562] Key type .llcrypt registered [13386.202978] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [13387.382203] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13387.463306] alg: No test for adler32 (adler32-zlib) [13388.916449] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [13389.206431] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [13390.955952] Key type lgssc registered [13392.771512] Lustre: Echo OBD driver; http://www.lustre.org/ [13403.910920] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13403.953172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13405.572047] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13412.716957] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [13414.885796] LustreError: 376395:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13414.925384] LustreError: 376395:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13420.019587] LustreError: 376394:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13424.892276] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13425.121206] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [13425.459383] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13431.304188] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [13440.145627] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13446.594359] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13456.037167] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13456.545863] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13465.497907] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [13474.883029] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13490.659223] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13490.663972] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13490.687600] Lustre: Skipped 1 previous similar message [13490.706773] Lustre: Skipped 1 previous similar message [13493.686760] Lustre: server umount lustre-OST0000 complete [13500.902385] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13500.920484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13500.927039] Lustre: Skipped 1 previous similar message [13500.951212] Lustre: Skipped 1 previous similar message [13506.018940] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13506.027306] Lustre: Skipped 1 previous similar message [13510.116314] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13510.130192] Lustre: Skipped 1 previous similar message [13512.160235] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13512.402363] Lustre: server umount lustre-MDT0000 complete [13515.235475] LustreError: 376394:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13515.247335] LustreError: 376394:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13516.319790] LustreError: 376324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451591 with bad export cookie 9833881574995050850 [13516.326166] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13516.329317] LustreError: 376324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13516.825218] Lustre: server umount lustre-MDT0001 complete [13527.600730] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [13530.737620] Key type lgssc unregistered [13531.136033] LNet: 382170:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13531.153569] LNetError: 382170:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13531.200177] LNet: Removed LNI 192.168.202.124@tcp [13532.103263] Key type .llcrypt unregistered [13532.105864] Key type ._llcrypt unregistered [13640.018484] Key type ._llcrypt registered [13640.022860] Key type .llcrypt registered [13640.122233] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [13640.418256] Key type .llcrypt unregistered [13640.420088] Key type ._llcrypt unregistered [13713.248517] Key type ._llcrypt registered [13713.264032] Key type .llcrypt registered [13713.421816] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [13713.762321] Key type .llcrypt unregistered [13713.767986] Key type ._llcrypt unregistered [13745.781763] Key type ._llcrypt registered [13745.784825] Key type .llcrypt registered [13745.866588] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [13746.137325] Key type .llcrypt unregistered [13746.139185] Key type ._llcrypt unregistered [13777.805696] Key type ._llcrypt registered [13777.807383] Key type .llcrypt registered [13777.914971] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [13778.152170] Key type .llcrypt unregistered [13778.155881] Key type ._llcrypt unregistered [13861.276620] Key type ._llcrypt registered [13861.280074] Key type .llcrypt registered [13861.348378] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [13861.750955] Key type .llcrypt unregistered [13861.754793] Key type ._llcrypt unregistered [13892.973945] Key type ._llcrypt registered [13892.985245] Key type .llcrypt registered [13893.145820] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [13893.491946] Key type .llcrypt unregistered [13893.494752] Key type ._llcrypt unregistered [13925.418840] Key type ._llcrypt registered [13925.423589] Key type .llcrypt registered [13925.520376] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [13925.806923] Key type .llcrypt unregistered [13925.810524] Key type ._llcrypt unregistered [13954.343063] Key type ._llcrypt registered [13954.344565] Key type .llcrypt registered [13954.437581] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [13954.669067] Key type .llcrypt unregistered [13954.671692] Key type ._llcrypt unregistered [13984.704933] Key type ._llcrypt registered [13984.707077] Key type .llcrypt registered [13984.810316] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [13985.120756] Key type .llcrypt unregistered [13985.124399] Key type ._llcrypt unregistered [14016.464736] Key type ._llcrypt registered [14016.471116] Key type .llcrypt registered [14016.590845] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [14016.856393] Key type .llcrypt unregistered [14016.859761] Key type ._llcrypt unregistered [14075.821923] Key type ._llcrypt registered [14075.829820] Key type .llcrypt registered [14075.916919] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [14076.151417] Key type .llcrypt unregistered [14076.154363] Key type ._llcrypt unregistered [14109.923955] Key type ._llcrypt registered [14109.926609] Key type .llcrypt registered [14110.027652] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [14110.285659] Key type .llcrypt unregistered [14110.288320] Key type ._llcrypt unregistered [14138.851530] Key type ._llcrypt registered [14138.853980] Key type .llcrypt registered [14138.969701] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [14139.266447] Key type .llcrypt unregistered [14139.269429] Key type ._llcrypt unregistered [14190.090187] Key type ._llcrypt registered [14190.092124] Key type .llcrypt registered [14190.192759] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_hostid [14204.745670] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [14205.964179] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14205.991993] alg: No test for adler32 (adler32-zlib) [14207.001394] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [14207.284905] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [14208.984202] Key type lgssc registered [14210.302747] Lustre: Echo OBD driver; http://www.lustre.org/ [14246.845177] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14246.898325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14248.228960] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [14248.283141] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [14248.417456] Lustre: tst250fs-MDT0000: new disk, initializing [14248.566511] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14248.621716] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14252.875532] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [14263.692309] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14263.793859] Lustre: 402686:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14263.844457] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [14263.848846] Lustre: Skipped 1 previous similar message [14263.906487] Lustre: tst250fs-MDT0001: new disk, initializing [14263.982066] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14264.029955] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14264.042116] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14267.867674] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [14272.179770] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [14278.267280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14278.491791] Lustre: tst250fs-OST0000: new disk, initializing [14278.495650] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [14278.501524] Lustre: 404315:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14278.578903] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14280.542991] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14280.552432] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14280.679327] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14284.019395] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [14293.397871] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14293.524326] Lustre: tst250fs-OST0001: new disk, initializing [14293.528479] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [14293.534622] Lustre: 405192:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14293.645270] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [14299.021855] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [14299.719275] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [14299.733339] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [14299.840800] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [14304.072664] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [14306.683830] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [14306.688429] Lustre: Skipped 2 previous similar messages [14325.732694] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14325.738421] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14325.744748] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14325.755641] Lustre: Skipped 3 previous similar messages [14325.778713] Lustre: Skipped 3 previous similar messages [14330.852092] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14330.857145] Lustre: Skipped 1 previous similar message [14331.971925] Lustre: server umount tst250fs-MDT0000 complete [14335.676256] LustreError: 402680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452410 with bad export cookie 7302267723188366304 [14335.692867] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14335.697859] LustreError: 402680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14335.974344] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14336.080634] Lustre: server umount tst250fs-MDT0001 complete [14342.904560] Lustre: server umount tst250fs-OST0000 complete [14347.567119] Lustre: server umount tst250fs-OST0001 complete [14366.973811] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_hostid [14377.010297] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [14411.134595] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 12:21:24 (1788452484) [14412.429603] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [14428.190772] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_hostid [14437.731472] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [14468.968273] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14137 sec ========= 12:22:22 (1788452542) [14470.682126] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 12:22:24 (1788452544) === [14473.789126] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 12:22:27 (1788452547) === [14512.363465] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [14515.177324] Key type lgssc unregistered [14515.504093] LNet: 414335:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14515.517100] LNetError: 414335:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14515.535493] LNet: Removed LNI 192.168.202.124@tcp [14516.358183] Key type .llcrypt unregistered [14516.361715] Key type ._llcrypt unregistered