[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 486870152 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001015] APIC: Switch to symmetric I/O mode setup [ 0.002368] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004014] kvm-guest: setup PV IPIs [ 0.007529] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010008] pid_max: default: 32768 minimum: 301 [ 0.011167] LSM: Security Framework initializing [ 0.012088] Yama: becoming mindful. [ 0.014047] SELinux: Initializing. [ 0.015112] *** VALIDATE selinux *** [ 0.024336] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.029269] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030171] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032167] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.034171] *** VALIDATE tmpfs *** [ 0.036157] *** VALIDATE proc *** [ 0.037295] *** VALIDATE cgroup *** [ 0.038017] *** VALIDATE cgroup2 *** [ 0.040341] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.042100] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.043009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.044032] Spectre V2 : User space: Vulnerable [ 0.045009] Speculative Store Bypass: Vulnerable [ 0.047994] debug: unmapping init [mem 0xffffffffa4259000-0xffffffffa4260fff] [ 0.050987] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.051771] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.052026] ... version: 2 [ 0.053013] ... bit width: 48 [ 0.054025] ... generic registers: 4 [ 0.055018] ... value mask: 0000ffffffffffff [ 0.056024] ... max period: 00007fffffffffff [ 0.057024] ... fixed-purpose events: 3 [ 0.058014] ... event mask: 000000070000000f [ 0.059305] rcu: Hierarchical SRCU implementation. [ 0.061605] smp: Bringing up secondary CPUs ... [ 0.062655] x86: Booting SMP configuration: [ 0.063045] .... node #0, CPUs: #1 #2 #3 [ 0.067114] smp: Brought up 1 node, 4 CPUs [ 0.069032] smpboot: Max logical packages: 1 [ 0.070017] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.132952] node 0 deferred pages initialised in 61ms [ 0.136231] devtmpfs: initialized [ 0.137231] x86/mm: Memory block size: 128MB [ 0.139245] gcov: version magic: 0x41383552 [ 0.141189] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.144179] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.146408] pinctrl core: initialized pinctrl subsystem [ 0.148298] [ 0.148962] ************************************************************* [ 0.151014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.153014] ** ** [ 0.155011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.157014] ** ** [ 0.159016] ** This means that this kernel is built to expose internal ** [ 0.161010] ** IOMMU data structures, which may compromise security on ** [ 0.162009] ** your system. ** [ 0.164007] ** ** [ 0.165011] ** If you see this message and you are not debugging the ** [ 0.167012] ** kernel, report this immediately to your vendor! ** [ 0.169017] ** ** [ 0.171019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.173019] ************************************************************* [ 0.175789] NET: Registered protocol family 16 [ 0.178523] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.181129] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.183072] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.187029] cpuidle: using governor menu [ 0.188428] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.191564] PCI: Using configuration type 1 for base access [ 0.194134] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.203052] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.204017] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.207138] cryptd: max_cpu_qlen set to 1000 [ 0.210205] ACPI: Added _OSI(Module Device) [ 0.212025] ACPI: Added _OSI(Processor Device) [ 0.212873] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.214031] ACPI: Added _OSI(Processor Aggregator Device) [ 0.220079] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.226362] ACPI: Interpreter enabled [ 0.227083] ACPI: PM: (supports S0 S3 S4 S5) [ 0.229017] ACPI: Using IOAPIC for interrupt routing [ 0.230128] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.235000] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.245289] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.248072] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.250026] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.253114] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.258403] acpiphp: Slot [2] registered [ 0.260140] acpiphp: Slot [5] registered [ 0.261193] acpiphp: Slot [6] registered [ 0.262070] acpiphp: Slot [7] registered [ 0.263115] acpiphp: Slot [8] registered [ 0.265156] acpiphp: Slot [9] registered [ 0.266121] acpiphp: Slot [10] registered [ 0.268200] acpiphp: Slot [3] registered [ 0.269131] acpiphp: Slot [4] registered [ 0.270133] acpiphp: Slot [11] registered [ 0.271120] acpiphp: Slot [12] registered [ 0.273129] acpiphp: Slot [13] registered [ 0.274102] acpiphp: Slot [14] registered [ 0.276153] acpiphp: Slot [15] registered [ 0.277170] acpiphp: Slot [16] registered [ 0.279137] acpiphp: Slot [17] registered [ 0.280109] acpiphp: Slot [18] registered [ 0.281082] acpiphp: Slot [19] registered [ 0.282147] acpiphp: Slot [20] registered [ 0.284317] acpiphp: Slot [21] registered [ 0.286235] acpiphp: Slot [22] registered [ 0.288134] acpiphp: Slot [23] registered [ 0.289420] acpiphp: Slot [24] registered [ 0.291124] acpiphp: Slot [25] registered [ 0.292188] acpiphp: Slot [26] registered [ 0.293123] acpiphp: Slot [27] registered [ 0.295124] acpiphp: Slot [28] registered [ 0.296135] acpiphp: Slot [29] registered [ 0.298176] acpiphp: Slot [30] registered [ 0.299184] acpiphp: Slot [31] registered [ 0.301095] PCI host bridge to bus 0000:00 [ 0.303025] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.305031] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.307057] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.311029] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.313036] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.316057] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.318191] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.321141] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.324367] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.334020] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.339069] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.341036] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.343024] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.344020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.347203] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.351000] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.352052] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.355803] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.362018] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.374017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.379028] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.385819] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.403024] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.417032] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.438027] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.449355] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.461030] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.470024] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.495024] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.510529] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.526027] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.537049] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.563031] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.580791] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.598029] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.610036] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.625026] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.637067] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.649026] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.663036] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.696047] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.707746] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.718031] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.729029] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.750025] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.763770] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.766716] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.769633] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.772692] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.775410] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.781152] iommu: Default domain type: Passthrough [ 0.782811] SCSI subsystem initialized [ 0.784252] ACPI: bus type USB registered [ 0.786238] usbcore: registered new interface driver usbfs [ 0.788183] usbcore: registered new interface driver hub [ 0.790300] usbcore: registered new device driver usb [ 0.793305] pps_core: LinuxPPS API ver. 1 registered [ 0.795034] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.798239] PTP clock support registered [ 0.800234] EDAC MC: Ver: 3.0.0 [ 0.802743] PCI: Using ACPI for IRQ routing [ 0.805279] NetLabel: Initializing [ 0.806016] NetLabel: domain hash size = 128 [ 0.808033] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.809132] NetLabel: unlabeled traffic allowed by default [ 0.812180] vgaarb: loaded [ 0.814436] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.816021] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.824246] clocksource: Switched to clocksource kvm-clock [ 0.942748] VFS: Disk quotas dquot_6.6.0 [ 0.944343] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.946909] *** VALIDATE ramfs *** [ 0.948292] *** VALIDATE hugetlbfs *** [ 0.949895] pnp: PnP ACPI init [ 0.953121] pnp: PnP ACPI: found 6 devices [ 0.970167] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.973373] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.975565] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.977742] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.980040] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.982399] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.985012] NET: Registered protocol family 2 [ 0.987180] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.992133] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.994417] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.998332] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.001653] TCP: Hash tables configured (established 65536 bind 65536) [ 1.004033] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.007224] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.010184] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.013501] NET: Registered protocol family 1 [ 1.016611] RPC: Registered named UNIX socket transport module. [ 1.019161] RPC: Registered udp transport module. [ 1.021385] RPC: Registered tcp transport module. [ 1.023601] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.026648] NET: Registered protocol family 44 [ 1.028890] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.031447] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.034038] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.036728] PCI: CLS 0 bytes, default 64 [ 1.038749] Unpacking initramfs... [ 2.531583] debug: unmapping init [mem 0xffff98527cc54000-0xffff98527ffbffff] [ 2.534056] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.535626] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.537383] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.045564] Initialise system trusted keyrings [ 3.046716] Key type blacklist registered [ 3.048202] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.057827] zbud: loaded [ 3.061477] *** VALIDATE nfs *** [ 3.062450] *** VALIDATE nfs4 *** [ 3.064250] pstore: using deflate compression [ 3.068193] Platform Keyring initialized [ 3.206922] NET: Registered protocol family 38 [ 3.210464] Key type asymmetric registered [ 3.211971] Asymmetric key parser 'x509' registered [ 3.213900] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.219137] io scheduler mq-deadline registered [ 3.221101] io scheduler kyber registered [ 3.222860] io scheduler bfq registered [ 3.225287] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.228572] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.231984] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.235332] ACPI: Power Button [PWRF] [ 3.243209] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.251154] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.271646] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.283576] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.309604] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.338383] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.369080] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.374803] Non-volatile memory driver v1.3 [ 3.376848] Linux agpgart interface v0.103 [ 3.416180] virtio_blk virtio1: [vda] 150424 512-byte logical blocks (77.0 MB/73.4 MiB) [ 3.419526] vda: detected capacity change from 0 to 77017088 [ 3.433187] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.435552] vdb: detected capacity change from 0 to 1073741824 [ 3.460057] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.462259] vdc: detected capacity change from 0 to 2621440000 [ 3.477786] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.479975] vdd: detected capacity change from 0 to 2621440000 [ 3.494786] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.497755] vde: detected capacity change from 0 to 4294967296 [ 3.533955] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.536149] vdf: detected capacity change from 0 to 4294967296 [ 3.543295] libphy: Fixed MDIO Bus: probed [ 3.555349] usbcore: registered new interface driver usbserial_generic [ 3.557772] usbserial: USB Serial support registered for generic [ 3.559587] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.562816] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.564224] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.567034] mousedev: PS/2 mouse device common for all mice [ 3.571121] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.572128] rtc_cmos 00:05: RTC can wake from S4 [ 3.580609] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.584399] rtc_cmos 00:05: registered as rtc0 [ 3.589474] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.590860] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.601488] intel_pstate: CPU model not supported [ 3.605269] hid: raw HID events driver (C) Jiri Kosina [ 3.607533] usbcore: registered new interface driver usbhid [ 3.609871] usbhid: USB HID core driver [ 3.611877] drop_monitor: Initializing network drop monitor service [ 3.615281] Initializing XFRM netlink socket [ 3.617955] NET: Registered protocol family 10 [ 3.621553] Segment Routing with IPv6 [ 3.623578] NET: Registered protocol family 17 [ 3.626292] mpls_gso: MPLS GSO support [ 3.634203] RAS: Correctable Errors collector initialized. [ 3.636588] AVX version of gcm_enc/dec engaged. [ 3.638304] AES CTR mode by8 optimization enabled [ 3.741441] sched_clock: Marking stable (3741338402, 0)->(4678760042, -937421640) [ 3.745890] registered taskstats version 1 [ 3.748450] Loading compiled-in X.509 certificates [ 3.751877] zswap: loaded using pool lzo/zbud [ 3.778939] Key type big_key registered [ 3.795119] Key type encrypted registered [ 3.797256] ima: No TPM chip found, activating TPM-bypass! [ 3.799671] ima: Allocated hash algorithm: sha1 [ 3.801146] ima: No architecture policies found [ 3.802647] evm: Initialising EVM extended attributes: [ 3.804116] evm: security.selinux [ 3.805560] evm: security.ima [ 3.806883] evm: security.capability [ 3.808399] evm: HMAC attrs: 0x1 [ 3.811128] rtc_cmos 00:05: setting system clock to 2026-09-05 02:15:51 UTC (1788574551) [ 3.817973] debug: unmapping init [mem 0xffffffffa5203000-0xffffffffa53fffff] [ 3.821077] debug: unmapping init [mem 0xffffffffa3f82000-0xffffffffa4258fff] [ 3.830117] Write protecting the kernel read-only data: 28672k [ 3.833863] debug: unmapping init [mem 0xffffffffa2603000-0xffffffffa27fffff] [ 3.836935] debug: unmapping init [mem 0xffffffffa2f14000-0xffffffffa2ffffff] [ 3.889165] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.898571] systemd[1]: Detected virtualization kvm. [ 3.900810] systemd[1]: Detected architecture x86-64. [ 3.902789] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.930701] systemd[1]: No hostname configured. [ 3.932655] systemd[1]: Set hostname to . [ 3.934872] random: systemd: uninitialized urandom read (16 bytes read) [ 3.937602] systemd[1]: Initializing machine ID from random generator. [ 4.095702] random: systemd: uninitialized urandom read (16 bytes read) [ 4.098161] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 4.103111] random: systemd: uninitialized urandom read (16 bytes read) [ 4.105557] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 4.108457] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Swap. [ OK ] Reached target Timers. Starting Apply Kernel Variables... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. [ OK ] Reached target Initrd Root Device. Starting Journal Service... [ OK ] Reached target Local File Systems. Starting Setup Virtual Console... Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.856266] device-mapper: uevent: version 1.0.3 [ 4.858463] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.620982] virtio_net virtio0 ens2: renamed from eth0 [ 5.644979] random: fast init done [ 5.675240] scsi host0: ata_piix [ 5.694630] scsi host1: ata_piix [ 5.696304] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.698954] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.332074] random: crng init done [ 10.335692] random: 7 urandom warning(s) missed due to ratelimiting [ 11.258168] dracut-initqueue[588]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 13.390492] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 16.811739] hrtimer: interrupt took 4642157 ns [ 16.995891] printk: systemd: 26 output lines suppressed due to ratelimiting [ 17.942914] SELinux: Disabled at runtime. [ 18.059795] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 18.081214] systemd[1]: Detected virtualization kvm. [ 18.085827] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 20.526063] systemd[1]: initrd-switch-root.service: Succeeded. [ 20.550232] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 20.568317] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 20.578098] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 20.584312] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 20.616573] systemd[1]: Starting Journal Service... Starting Journal Service... [ 20.646135] systemd[1]: Created slice system-getty.slice. [ OK ] Created slice system-getty.slice. [ OK ] Reached target rpc_pipefs.target. Mounting Kernel Debug File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Control Socket. [ 20.876726] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. Starting Remount Root and Kernel File Systems... Mounting Huge Pages File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Initrd Root File System. Mounting POSIX Message Queue File System... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 22.612483] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 24.094762] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 24.303455] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 25.051592] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 25.132790] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (9s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ ***] A start job is running for Configur…only root support (11s / no limit)[ 31.631222] Key type dns_resolver registered [ **] A start job is running for Configur…only root support (11s / no limit) [ *] A start job is running for Configur…only root support (12s / no limit)[ 32.503416] NFS: Registering the id_resolver key type [ 32.522333] Key type id_resolver registered [ 32.527603] Key type id_legacy registered [ **] A start job is running for Configur…only root support (12s / no limit) [ ***] A start job is running for Configur…only root support (12s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Load/Save Random Seed. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Login Service. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg133-server login: [ 106.598956] libcfs: loading out-of-tree module taints kernel. [ 106.700455] Key type ._llcrypt registered [ 106.702111] Key type .llcrypt registered [ 106.810907] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_hostid [ 131.453922] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 133.986281] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 134.034417] alg: No test for adler32 (adler32-zlib) [ 135.850278] Lustre: Lustre: Build Version: 2.17.58_3_g4a08d7b [ 137.428054] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 139.274653] Key type lgssc registered [ 141.851562] Lustre: Echo OBD driver; http://www.lustre.org/ [ 168.252562] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 226.580101] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 244.846427] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 244.901870] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 246.428392] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 246.536847] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 246.749838] Lustre: lustre-MDT0000: new disk, initializing [ 246.967676] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 246.999199] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 253.922899] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 273.409865] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 273.656482] Lustre: 6514:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 273.754290] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 273.764809] Lustre: Skipped 1 previous similar message [ 274.029944] Lustre: lustre-MDT0001: new disk, initializing [ 274.217310] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 274.302712] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 274.332898] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 280.985186] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 287.916521] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 300.576869] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 301.211401] Lustre: lustre-OST0000: new disk, initializing [ 301.222084] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 301.234720] Lustre: 8453:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 301.416035] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 309.825050] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 309.838796] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 310.033605] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 310.456375] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 331.841155] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 332.021735] Lustre: lustre-OST0001: new disk, initializing [ 332.026965] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 332.032742] Lustre: 9530:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 332.140483] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 337.935646] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 337.945629] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 338.089606] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 341.420061] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 356.919739] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 365.641328] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 373.814302] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing check_logdir /tmp/testlogs/ [ 380.155571] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing yml_node [ 386.108942] Lustre: DEBUG MARKER: Client: 2.17.58.3 [ 390.039505] Lustre: DEBUG MARKER: MDS: 2.17.58.3 [ 393.497049] Lustre: DEBUG MARKER: OSS: 2.17.58.3 [ 395.442588] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Sep 4 22:22:20 EDT 2026 [ 418.154219] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 420.402641] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 440.810188] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 440.817310] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 440.827246] Lustre: Skipped 3 previous similar messages [ 440.849618] Lustre: Skipped 3 previous similar messages [ 445.114197] Lustre: server umount lustre-MDT0000 complete [ 449.837089] LustreError: 6506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788574997 with bad export cookie 7250669345501665962 [ 449.848389] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 449.852753] LustreError: 6506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 451.041101] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 451.041958] LustreError: 9711:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 451.065983] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 451.088589] LustreError: 9711:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 451.112697] Lustre: Skipped 1 previous similar message [ 454.056516] LustreError: 6522:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 454.060294] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 454.076643] LustreError: 6522:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 454.099823] Lustre: Skipped 1 previous similar message [ 459.235524] LustreError: 9711:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 459.237862] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 459.253473] LustreError: 9711:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 459.270298] Lustre: Skipped 1 previous similar message [ 464.353379] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 464.744738] Lustre: server umount lustre-MDT0001 complete [ 470.938772] Lustre: server umount lustre-OST0000 complete [ 476.793716] Lustre: server umount lustre-OST0001 complete [ 485.745576] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_hostid [ 498.013961] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 540.213486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 540.476347] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 540.513733] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 540.652201] Lustre: lustre-MDT0000: new disk, initializing [ 540.851221] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 540.907384] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 546.447530] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 558.730024] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 558.867130] Lustre: 15091:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 558.937807] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 558.942387] Lustre: Skipped 1 previous similar message [ 559.033213] Lustre: lustre-MDT0001: new disk, initializing [ 559.123705] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 559.180725] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 559.186164] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 564.588185] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 575.486607] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 582.010716] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 589.039323] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 589.266700] Lustre: lustre-OST0000: new disk, initializing [ 589.272625] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 589.285333] Lustre: 17061:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 589.386562] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 590.993921] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 591.006130] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 591.172243] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 597.626822] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 611.102644] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 617.978788] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 618.378683] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 623.308065] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 623.590808] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 627.180776] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 627.185181] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 627.189728] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 627.189743] Lustre: Skipped 2 previous similar messages [ 627.193485] Lustre: Skipped 2 previous similar messages [ 627.248080] LustreError: Skipped 1 previous similar message [ 632.299324] LustreError: 17046:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 632.320708] LustreError: 17046:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 632.459408] Lustre: server umount lustre-OST0000 complete [ 637.410700] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 637.426388] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 637.427796] Lustre: Skipped 1 previous similar message [ 637.443365] Lustre: Skipped 2 previous similar messages [ 643.587792] Lustre: server umount lustre-MDT0000 complete [ 647.654502] LustreError: 15105:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 647.676323] LustreError: 15105:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 648.176037] LustreError: 15085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788575195 with bad export cookie 7250669345501667495 [ 648.203498] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 648.633342] Lustre: server umount lustre-MDT0001 complete [ 655.617807] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 22:26:41 (1788575201) [ 676.799652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 677.301204] LustreError: 19938:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 677.414974] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 683.117796] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 693.635756] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 700.416391] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 708.118850] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 715.277941] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 717.795419] Lustre: Failing over lustre-MDT0000 [ 718.110372] Lustre: server umount lustre-MDT0000 complete [ 718.818410] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 718.828413] LustreError: 19935:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 718.834153] Lustre: Skipped 1 previous similar message [ 718.872301] LustreError: 19935:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 735.202048] Lustre: 3645:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788575266/real 1788575266] req@ffff9852fc7f4380 x1875456489077376/t0(0) o400->MGC192.168.201.133@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788575282 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 735.248647] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 739.333050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 744.545346] LustreError: 3643:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9852fc7f5180 x1875456489080320/t0(0) o250->MGC192.168.201.133@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 744.902276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 744.911806] Lustre: Skipped 1 previous similar message [ 745.006515] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 750.050518] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 750.071794] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 750.106836] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 750.597725] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 761.521244] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 768.493097] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 768.499167] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 768.519570] Lustre: Skipped 1 previous similar message [ 768.529768] Lustre: Skipped 3 previous similar messages [ 772.190159] Lustre: server umount lustre-MDT0000 complete [ 773.602115] LustreError: 19939:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 773.635440] LustreError: 19939:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 15 previous similar messages [ 776.405083] LustreError: 20673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788575324 with bad export cookie 7250669345501668909 [ 776.406101] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 776.426688] LustreError: 20673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 777.076421] Lustre: server umount lustre-MDT0001 complete [ 787.812419] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 794.754077] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 805.410185] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 811.420915] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 819.998176] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 826.637441] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 829.126309] Lustre: Failing over lustre-MDT0000 [ 829.402218] Lustre: server umount lustre-MDT0000 complete [ 831.202698] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 841.702180] LustreError: 24221:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 841.727732] LustreError: 24221:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 846.752233] Lustre: 3647:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788575378/real 1788575378] req@ffff9852fc791f80 x1875456489107840/t0(0) o400->MGC192.168.201.133@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788575394 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 846.786873] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 849.560691] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 857.057027] LustreError: 3643:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9852df7b1180 x1875456489110400/t0(0) o250->MGC192.168.201.133@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 857.613204] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 862.637946] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 862.697509] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 862.706153] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 862.716789] Lustre: Skipped 1 previous similar message [ 862.730878] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 872.703413] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 878.052812] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 878.065853] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 878.092291] Lustre: Skipped 1 previous similar message [ 878.110490] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 891.360435] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 891.761146] Lustre: server umount lustre-MDT0000 complete [ 896.635270] LustreError: 25940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788575444 with bad export cookie 7250669345501669847 [ 896.652063] LustreError: 25940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 896.653396] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 897.081798] Lustre: server umount lustre-MDT0001 complete [ 908.419361] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 22:30:53 (1788575453) [ 931.289446] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_hostid [ 944.415914] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 990.801499] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 1003.601419] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1003.883232] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1003.908764] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1004.069887] Lustre: lustre-MDT0000: new disk, initializing [ 1004.132912] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1004.136245] Lustre: Skipped 3 previous similar messages [ 1004.154221] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1009.809331] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 1023.177543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1023.465809] Lustre: 29658:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1023.713941] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1029.555533] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 1041.382672] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1047.586211] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1054.759326] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1055.153209] Lustre: 31628:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1056.881916] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1056.935762] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 1057.062387] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1057.088458] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 1062.016721] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 1072.388462] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1078.656362] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 1103.329967] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1103.346800] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1103.365243] Lustre: Skipped 1 previous similar message [ 1103.374179] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1103.376883] Lustre: Skipped 4 previous similar messages [ 1105.983911] Lustre: server umount lustre-OST0000 complete [ 1111.011442] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1111.023884] LustreError: Skipped 1 previous similar message [ 1116.501844] Lustre: server umount lustre-MDT0000 complete [ 1118.692155] LustreError: 29666:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1118.715257] LustreError: 29666:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 1120.043931] LustreError: 29653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788575667 with bad export cookie 7250669345501670253 [ 1120.049737] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1120.058898] LustreError: 29653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1120.442211] Lustre: server umount lustre-MDT0001 complete [ 1132.896508] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 1136.575101] Key type lgssc unregistered [ 1136.965913] LNet: 33606:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1136.972714] LNetError: 33606:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1136.985586] LNet: Removed LNI 192.168.201.133@tcp [ 1138.126347] Key type .llcrypt unregistered [ 1138.128846] Key type ._llcrypt unregistered [ 1165.386958] Key type ._llcrypt registered [ 1165.389703] Key type .llcrypt registered [ 1165.496815] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 1166.969351] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1167.157288] alg: No test for adler32 (adler32-zlib) [ 1168.375905] Lustre: Lustre: Build Version: 2.17.58_3_g4a08d7b [ 1168.664043] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 1170.410330] Key type lgssc registered [ 1172.063075] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1180.730749] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1180.742479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1182.245917] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1187.336416] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 1191.400551] LustreError: 35017:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1191.434261] LustreError: 35017:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1195.832615] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1196.034969] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1200.510864] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 1208.001605] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1214.175991] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1221.251065] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1221.779785] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1227.032367] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1228.475906] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 1237.390865] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1241.268122] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1256.123318] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 22:36:40 (1788575800) [ 1258.310319] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1260.403707] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1268.290084] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 22:36:53 (1788575813) [ 1276.798889] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 22:37:02 (1788575822) [ 1277.729475] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1287.036425] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1287.042033] Lustre: Skipped 2 previous similar messages [ 1288.605975] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 22:37:14 (1788575834) [ 1292.547816] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1292.553581] Lustre: Skipped 3 previous similar messages [ 1312.586923] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1312.595610] Lustre: Skipped 6 previous similar messages [ 1315.775324] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 22:37:41 (1788575861) [ 1318.306464] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1321.708319] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1329.352116] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 22:37:55 (1788575875) [ 1331.146660] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1331.156820] Lustre: Skipped 3 previous similar messages [ 1345.039870] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 22:38:10 (1788575890) [ 1360.354547] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1360.368082] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1360.391698] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1364.464399] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1364.464819] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1364.464913] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1364.486786] Lustre: Skipped 1 previous similar message [ 1365.858394] Lustre: server umount lustre-OST0000 complete [ 1370.084117] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1370.093679] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1370.115089] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1374.692044] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1374.695523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1374.745584] Lustre: Skipped 1 previous similar message [ 1376.125124] Lustre: server umount lustre-MDT0000 complete [ 1379.786109] LustreError: 35000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788575927 with bad export cookie 11564359497678991562 [ 1379.794103] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1379.801387] LustreError: 35000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1380.157817] Lustre: server umount lustre-MDT0001 complete [ 1391.512971] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 1393.973504] Key type lgssc unregistered [ 1394.262305] LNet: 42871:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1394.270271] LNetError: 42871:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1394.287688] LNet: Removed LNI 192.168.201.133@tcp [ 1395.319957] Key type .llcrypt unregistered [ 1395.321823] Key type ._llcrypt unregistered [ 1414.331983] Key type ._llcrypt registered [ 1414.336433] Key type .llcrypt registered [ 1414.489737] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_hostid [ 1431.514432] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 1432.856152] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1432.883838] alg: No test for adler32 (adler32-zlib) [ 1434.004645] Lustre: Lustre: Build Version: 2.17.58_3_g4a08d7b [ 1434.287927] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 1436.072390] Key type lgssc registered [ 1437.804530] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1477.967205] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1477.993351] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1479.344826] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1479.421958] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1479.591384] Lustre: lustre-MDT0000: new disk, initializing [ 1479.710102] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1479.737748] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1484.435802] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 1496.780852] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1497.007789] Lustre: 46021:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1497.080860] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1497.098942] Lustre: Skipped 1 previous similar message [ 1497.266067] Lustre: lustre-MDT0001: new disk, initializing [ 1497.451347] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1497.504469] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1497.530038] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1503.109235] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 1517.180688] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1523.983416] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1535.175283] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1535.662273] Lustre: lustre-OST0000: new disk, initializing [ 1535.670621] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1535.683154] Lustre: 47996:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1535.854356] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1543.280946] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 1544.851918] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1544.862807] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1545.064553] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1554.456990] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1559.953224] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1560.211779] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1564.881503] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1565.073733] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1570.787098] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1570.789967] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1570.799723] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1570.802430] Lustre: Skipped 1 previous similar message [ 1570.832975] Lustre: Skipped 1 previous similar message [ 1573.322753] Lustre: server umount lustre-OST0000 complete [ 1579.489942] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1579.498332] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1579.512241] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1581.034828] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1581.049479] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1586.147821] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1586.153539] Lustre: Skipped 2 previous similar messages [ 1591.265739] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1591.278282] Lustre: Skipped 1 previous similar message [ 1592.288122] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1592.664976] Lustre: server umount lustre-MDT0000 complete [ 1596.663317] LustreError: 46016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788576144 with bad export cookie 5978673057500223931 [ 1596.671900] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1596.976748] Lustre: server umount lustre-MDT0001 complete [ 1606.762743] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 22:42:32 (1788576152) [ 1612.953131] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1613.399642] LustreError: 50456:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1613.421547] LustreError: 50456:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1613.501650] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1617.791095] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 1618.914687] LustreError: 50457:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1624.036455] LustreError: 50456:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1626.639032] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1626.931763] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1631.891617] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 1639.219282] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1645.160651] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1652.479227] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1652.725461] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1658.769122] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 1665.389699] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1678.636745] LustreError: 53006:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1679.583127] Lustre: Modifying parameter general.timeout=1 in log params [ 1680.608901] Lustre: Modifying parameter general.timeout=2 in log params [ 1683.076457] Lustre: Modifying parameter general.timeout=5 in log params [ 1683.080179] Lustre: Skipped 2 previous similar messages [ 1687.730276] Lustre: Modifying parameter general.timeout=11 in log params [ 1687.732825] Lustre: Skipped 5 previous similar messages [ 1696.230621] Lustre: Modifying parameter general.timeout=22 in log params [ 1696.234503] Lustre: Skipped 10 previous similar messages [ 1712.718898] Lustre: Modifying parameter general.timeout=41 in log params [ 1712.724163] Lustre: Skipped 18 previous similar messages [ 1744.818857] Lustre: Modifying parameter general.timeout=73 in log params [ 1744.822127] Lustre: Skipped 31 previous similar messages [ 1809.308299] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 22:45:54 (1788576354) [ 1810.476807] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_policies=tbf in log params [ 1810.483071] Lustre: Skipped 57 previous similar messages [ 1811.871073] Lustre: 59950:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1813.620605] Lustre: 60048:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1815.790212] Lustre: 60162:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1821.152878] Lustre: 60458:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1821.167907] Lustre: 60458:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1849.292384] Lustre: 61829:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1849.314961] LustreError: 61829:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1853.333343] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 22:46:38 (1788576398) [ 1867.753833] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1867.765789] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1867.786139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1872.157426] Lustre: server umount lustre-MDT0000 complete [ 1875.187310] LustreError: 50437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788576422 with bad export cookie 5978673057500224813 [ 1875.192026] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1875.207969] LustreError: 50437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 1875.506149] Lustre: server umount lustre-MDT0001 complete [ 1889.296923] Lustre: server umount lustre-OST0000 complete [ 1916.939990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1917.512139] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1918.101351] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1922.756873] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 1931.524235] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1931.649975] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1931.670717] Lustre: 64358:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1937.073942] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 1945.724380] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1951.649818] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1958.427377] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1958.604206] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1958.894927] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1958.903199] Lustre: Skipped 1 previous similar message [ 1965.525720] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 1973.699259] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1975.516518] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1975.524491] Lustre: Skipped 22 previous similar messages [ 1975.566297] Lustre: 66926:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1983.457569] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1983.466751] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1983.483772] Lustre: Skipped 2 previous similar messages [ 1983.494545] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1983.498031] Lustre: Skipped 3 previous similar messages [ 1988.426758] Lustre: server umount lustre-OST0000 complete [ 1993.186229] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1993.194345] LustreError: Skipped 1 previous similar message [ 1993.199612] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1993.209917] Lustre: Skipped 1 previous similar message [ 1998.570148] Lustre: server umount lustre-MDT0000 complete [ 1999.334366] LustreError: 64368:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1999.366769] LustreError: 64368:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2001.705959] LustreError: 64351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788576549 with bad export cookie 5978673057500227900 [ 2001.713500] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2001.717394] LustreError: 64351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2002.098652] Lustre: server umount lustre-MDT0001 complete [ 2013.651602] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 2016.691409] Key type lgssc unregistered [ 2017.020782] LNet: 68136:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2017.029256] LNetError: 68136:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2017.044110] LNet: Removed LNI 192.168.201.133@tcp [ 2017.983089] Key type .llcrypt unregistered [ 2017.984486] Key type ._llcrypt unregistered [ 2045.365766] Key type ._llcrypt registered [ 2045.368655] Key type .llcrypt registered [ 2045.502879] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 2046.679204] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2046.827959] alg: No test for adler32 (adler32-zlib) [ 2047.930868] Lustre: Lustre: Build Version: 2.17.58_3_g4a08d7b [ 2048.199772] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 2049.920200] Key type lgssc registered [ 2051.293542] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2059.013376] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2059.057371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2060.812711] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2065.818224] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2071.024587] LustreError: 69549:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2071.038305] LustreError: 69549:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2074.747094] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2075.399643] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2080.352758] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2087.995979] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2093.419226] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2101.345409] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2102.090677] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2108.824241] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2115.575743] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2126.598783] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2135.011227] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2135.014685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2135.021472] Lustre: Skipped 2 previous similar messages [ 2135.029674] Lustre: Skipped 2 previous similar messages [ 2140.132070] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2140.137700] Lustre: Skipped 1 previous similar message [ 2141.281103] Lustre: server umount lustre-MDT0000 complete [ 2145.180159] LustreError: 69530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788576692 with bad export cookie 8069974574239366763 [ 2145.189037] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2145.199283] LustreError: 69530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2145.250178] LustreError: 69546:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2145.250268] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2145.276988] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2145.279905] Lustre: Skipped 1 previous similar message [ 2145.520547] Lustre: server umount lustre-MDT0001 complete [ 2151.532432] Lustre: server umount lustre-OST0000 complete [ 2177.381093] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2177.839593] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2177.887208] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2178.415051] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2184.036570] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2193.484539] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2193.579609] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2193.599432] Lustre: 74420:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2193.992280] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2199.807351] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2208.646258] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2215.284380] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2223.446427] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2223.824383] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2224.467699] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2231.024423] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2238.803166] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2240.767664] Lustre: 76991:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2242.719326] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2248.674459] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2248.688427] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2248.704768] Lustre: Skipped 1 previous similar message [ 2253.798367] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2253.812906] Lustre: Skipped 2 previous similar messages [ 2254.965425] Lustre: server umount lustre-OST0000 complete [ 2260.458633] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2260.464741] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2264.049281] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2264.059956] Lustre: Skipped 2 previous similar messages [ 2265.773906] Lustre: server umount lustre-MDT0000 complete [ 2269.153748] LustreError: 74430:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2269.179599] LustreError: 74430:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2271.014559] LustreError: 74412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788576818 with bad export cookie 8069974574239367596 [ 2271.024852] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2271.027456] LustreError: 74412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2271.420323] Lustre: server umount lustre-MDT0001 complete [ 2284.212930] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 2287.470512] Key type lgssc unregistered [ 2287.875325] LNet: 78202:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2287.893125] LNetError: 78202:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2287.915499] LNet: Removed LNI 192.168.201.133@tcp [ 2288.982359] Key type .llcrypt unregistered [ 2288.984198] Key type ._llcrypt unregistered [ 2313.425145] Key type ._llcrypt registered [ 2313.427814] Key type .llcrypt registered [ 2313.514689] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 2314.387052] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2314.495783] alg: No test for adler32 (adler32-zlib) [ 2315.641680] Lustre: Lustre: Build Version: 2.17.58_3_g4a08d7b [ 2315.865213] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 2317.520189] Key type lgssc registered [ 2318.661356] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2325.681973] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2325.720814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2327.181397] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2332.578301] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2336.746884] LustreError: 79613:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2336.786858] LustreError: 79613:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2340.594962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2341.010066] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2344.872783] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2351.201789] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2356.631345] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2363.262484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2363.590443] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2369.750356] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2376.794735] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2383.554947] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2384.318772] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2385.877222] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2385.884902] Lustre: Skipped 1 previous similar message [ 2387.984611] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2388.002690] Lustre: Skipped 2 previous similar messages [ 2392.613190] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2392.615519] Lustre: Skipped 5 previous similar messages [ 2401.026963] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2401.033716] Lustre: Skipped 8 previous similar messages [ 2417.314380] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2417.318334] Lustre: Skipped 18 previous similar messages [ 2449.378425] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2449.388883] Lustre: Skipped 32 previous similar messages [ 2480.083655] Lustre: 87710:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2482.509562] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 22:57:08 (1788577028) [ 2484.070190] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2485.680048] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 22:57:11 (1788577031) [ 2489.833604] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2489.843676] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2489.850170] Lustre: Skipped 1 previous similar message [ 2489.867463] Lustre: Skipped 1 previous similar message [ 2494.978374] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2495.450691] Lustre: server umount lustre-OST0000 complete [ 2499.563740] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2499.571633] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2499.587343] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2499.592618] Lustre: Skipped 1 previous similar message [ 2505.189209] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2505.195460] Lustre: Skipped 3 previous similar messages [ 2510.306092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2510.311546] Lustre: Skipped 1 previous similar message [ 2513.376816] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2515.425748] LustreError: 79613:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2515.436981] LustreError: 79613:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2515.645686] Lustre: server umount lustre-MDT0000 complete [ 2519.032771] LustreError: 79592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788577066 with bad export cookie 16433248392183825092 [ 2519.039561] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2519.040263] LustreError: 79592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2519.463540] Lustre: server umount lustre-MDT0001 complete [ 2530.847538] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 2533.760318] Key type lgssc unregistered [ 2534.026096] LNet: 88959:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2534.030189] LNetError: 88959:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2534.063477] LNet: Removed LNI 192.168.201.133@tcp [ 2535.025059] Key type .llcrypt unregistered [ 2535.030917] Key type ._llcrypt unregistered [ 2540.228790] Key type ._llcrypt registered [ 2540.230951] Key type .llcrypt registered [ 2540.352862] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2544.999703] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules [ 2546.231621] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2546.266472] alg: No test for adler32 (adler32-zlib) [ 2547.433501] Lustre: Lustre: Build Version: 2.17.58_3_g4a08d7b [ 2547.657125] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 2549.377322] Key type lgssc registered [ 2551.163889] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2552.252076] LustreError: 90162:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2567.942968] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 2573.654011] LustreError: 90162:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2577.259851] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2577.279866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2578.857069] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2583.712409] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2589.155805] LustreError: 91184:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2589.178697] LustreError: 91184:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2590.184319] LustreError: 91183:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2591.882027] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 22:58:57 (1788577137) [ 2595.298364] LustreError: 91184:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2596.082059] LustreError: 91215:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2596.102620] LustreError: 91215:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2596.113804] LustreError: 91215:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 17, retries 0, failed: rc = -5 [ 2596.417895] Lustre: server umount lustre-MDT0000 complete [ 2609.624065] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 2612.344694] Key type lgssc unregistered [ 2612.672374] LNet: 92707:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2612.684013] LNetError: 92707:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2612.709899] LNet: Removed LNI 192.168.201.133@tcp [ 2613.394378] Key type .llcrypt unregistered [ 2613.397349] Key type ._llcrypt unregistered [ 2629.614567] Key type ._llcrypt registered [ 2629.618432] Key type .llcrypt registered [ 2629.721950] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 2630.963531] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2631.194777] alg: No test for adler32 (adler32-zlib) [ 2632.401732] Lustre: Lustre: Build Version: 2.17.58_3_g4a08d7b [ 2632.627252] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 2634.386952] Key type lgssc registered [ 2635.619787] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2642.727704] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2642.746684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2644.355954] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2649.824379] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2653.691323] LustreError: 93918:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2653.704992] LustreError: 93918:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2658.792737] LustreError: 93919:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2659.366407] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2660.292343] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2665.416039] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2674.913524] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2681.684831] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2688.439444] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2689.368758] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2695.484492] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2703.663887] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2750.636676] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 23:01:36 (1788577296) [ 2752.466917] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2754.083248] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 23:01:39 (1788577299) [ 2761.190888] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2761.203556] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2761.207908] Lustre: Skipped 1 previous similar message [ 2761.232121] Lustre: Skipped 2 previous similar messages [ 2765.165787] Lustre: server umount lustre-MDT0000 complete [ 2766.320170] LustreError: 95359:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2766.345211] LustreError: 95359:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2768.297166] LustreError: 93900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788577315 with bad export cookie 18226594845387674749 [ 2768.305306] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2768.502170] Lustre: server umount lustre-MDT0001 complete [ 2782.310739] Lustre: server umount lustre-OST0000 complete [ 2794.418679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2794.710979] LustreError: 98210:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2794.816794] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2799.309266] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2800.100575] LustreError: 98211:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2807.526302] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2808.262329] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2813.501419] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2819.856651] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2824.668031] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2841.141713] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2841.328714] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2841.340323] LustreError: 98200:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2841.372258] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2841.378216] LustreError: 100293:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -98 [ 2841.384910] LustreError: 100293:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2841.391161] LustreError: 100293:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2841.435695] Lustre: server umount lustre-OST0000 complete [ 2841.437671] LustreError: 100293:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2850.214326] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2850.487620] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2850.504803] LustreError: 98199:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2850.537164] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2850.551171] LustreError: 100756:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -98 [ 2850.561161] LustreError: 100756:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2850.569211] LustreError: 100756:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2850.707857] Lustre: server umount lustre-OST0000 complete [ 2850.711895] LustreError: 100756:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2860.531909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2860.699867] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2860.703642] Lustre: Found index 0 for lustre-OST0000, updating log [ 2860.708991] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2860.804441] Lustre: lustre-OST0000: new disk, initializing [ 2860.809617] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2860.814982] Lustre: 101294:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2861.177179] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2868.184832] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2872.287945] Lustre: Failing over lustre-OST0000 [ 2872.431139] Lustre: server umount lustre-OST0000 complete [ 2877.356696] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2877.381604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2877.394518] Lustre: Skipped 2 previous similar messages [ 2877.415666] Lustre: Skipped 1 previous similar message [ 2882.535210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2882.552587] Lustre: Skipped 1 previous similar message [ 2886.735450] Lustre: server umount lustre-MDT0000 complete [ 2887.651851] LustreError: 98211:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2887.675334] LustreError: 98211:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2890.912257] LustreError: 98190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788577438 with bad export cookie 18226594845387677493 [ 2890.914843] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2890.921265] LustreError: 98190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 2891.361702] Lustre: server umount lustre-MDT0001 complete [ 2900.561501] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 23:04:06 (1788577446) [ 2917.728152] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 2926.749206] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2927.037236] LustreError: 103640:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2927.125538] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2931.209260] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2938.891682] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2943.628766] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2946.541738] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2952.809269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2953.404665] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2959.908653] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2963.619231] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2963.640459] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 2963.657879] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2968.362750] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2968.441175] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2968.557139] Lustre: lustre-OST0001: new disk, initializing [ 2968.562561] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2968.568859] Lustre: 105960:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2968.895408] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:54 [ 2968.918371] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2968.942596] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2968.952940] Lustre: Skipped 2 previous similar messages [ 2973.032398] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 1:00 [ 2973.056920] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2973.074446] Lustre: lustre-OST0000: Recovery over after 0:10, of 1 clients 1 recovered and 0 were evicted. [ 2973.078161] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2973.086024] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2976.188554] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2976.973870] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2976.995113] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2977.254968] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2984.454602] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2992.771480] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2997.751700] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2997.790211] Lustre: Skipped 1 previous similar message [ 2997.794483] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2997.823525] Lustre: Skipped 2 previous similar messages [ 3002.853653] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3002.860763] Lustre: Skipped 3 previous similar messages [ 3003.918167] Lustre: server umount lustre-MDT0001 complete [ 3004.911459] LustreError: 106358:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3004.942544] LustreError: 106358:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3014.492749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3014.720486] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3014.736479] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3014.824951] Lustre: 103627:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3015.164693] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.33@tcp (not set up) [ 3016.697798] LustreError: 107546:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 3016.704752] LustreError: 107546:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 3016.715020] LustreError: 103632:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 3016.822773] LustreError: 107553:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.133@tcp: cfg command failed: rc = -17 [ 3016.832520] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3016.832520] [ 3020.052250] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3020.795369] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3029.266970] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 23:06:14 (1788577574) [ 3045.012815] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 3061.227934] Lustre: 109576:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3083.118907] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3086.288070] Lustre: 110712:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3163.624147] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3163.636337] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3163.660797] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3169.000353] Lustre: server umount lustre-MDT0000 complete [ 3169.248989] LustreError: 103641:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3169.259742] LustreError: 103641:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 3172.490700] LustreError: 104385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788577720 with bad export cookie 18226594845387678494 [ 3172.494522] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3172.502176] LustreError: 104385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3173.024446] Lustre: server umount lustre-MDT0001 complete [ 3187.500406] Lustre: server umount lustre-OST0000 complete [ 3191.328116] Lustre: 93228:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788577722/real 1788577722] req@ffff9851cabe3b80 x1875459105446784/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788577738 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3191.363902] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3191.382404] Lustre: Skipped 3 previous similar messages [ 3191.876256] Lustre: server umount lustre-OST0001 complete [ 3197.755526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3207.459804] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3215.710492] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3224.702360] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3243.812142] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 3253.760456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3253.787674] Lustre: lustre-MDT0000: reset Object Index mappings [ 3254.406524] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3254.412806] Lustre: Skipped 1 previous similar message [ 3258.815212] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3266.835710] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3266.860612] Lustre: lustre-MDT0001: reset Object Index mappings [ 3272.463697] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3275.402644] Lustre: 115814:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3281.600257] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3288.751609] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3291.539604] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:200 to 0x280000400:225) [ 3296.044674] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3298.727848] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:381 to 0x280000401:418) [ 3298.743883] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 3298.788985] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 3302.151873] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3309.113433] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3312.228585] Lustre: 117692:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3357.361384] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 23:11:43 (1788577903) [ 3384.814714] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3384.818975] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3384.829284] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3384.834137] Lustre: Skipped 4 previous similar messages [ 3389.119827] Lustre: server umount lustre-MDT0000 complete [ 3390.956279] LustreError: 114664:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3390.963857] LustreError: 114664:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 3392.430250] LustreError: 114637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788577940 with bad export cookie 18226594845387725548 [ 3392.440364] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3392.443760] LustreError: 114637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3399.247731] Lustre: server umount lustre-MDT0001 complete [ 3417.056095] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3417.188093] Lustre: server umount lustre-OST0000 complete [ 3427.052217] Lustre: server umount lustre-OST0001 complete [ 3433.704989] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_hostid [ 3442.047836] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 3478.725625] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3479.021660] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3479.073216] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3479.172640] Lustre: lustre-MDT0000: new disk, initializing [ 3479.268949] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3479.274175] Lustre: Skipped 3 previous similar messages [ 3479.295708] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3483.913235] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3491.306416] Lustre: server umount lustre-MDT0000 complete [ 3500.459750] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3500.705917] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3500.719520] Lustre: 121961:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3500.757554] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3505.892786] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3511.025785] Lustre: server umount lustre-MDT0000 complete [ 3520.662425] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 23:14:26 (1788578066) [ 3522.334968] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3524.659173] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 23:14:29 (1788578069) [ 3545.070976] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 3556.651656] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3562.596400] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3571.696761] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3571.828359] Lustre: 124013:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3571.882561] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3571.885705] Lustre: Skipped 1 previous similar message [ 3571.941265] Lustre: lustre-MDT0001: new disk, initializing [ 3572.049099] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3572.062930] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3576.326582] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3580.947509] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3588.372862] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3588.911629] Lustre: lustre-OST0000: new disk, initializing [ 3588.920405] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3588.937626] Lustre: 125592:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3590.015476] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3590.038745] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3590.200451] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3597.982493] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3608.493987] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3608.671882] Lustre: lustre-OST0001: new disk, initializing [ 3608.674817] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3608.679599] Lustre: 126463:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3610.676648] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3610.680931] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3610.818612] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3614.176441] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3625.192769] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3629.448546] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4003.224927] Lustre: 132874:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4003.240488] Lustre: 132874:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4003.255860] Lustre: 132874:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4003.267880] Lustre: 132874:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4003.281838] Lustre: 132874:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4003.294670] Lustre: 132874:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4282.782624] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 23:27:07 (1788578827) [ 4299.233843] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4299.255870] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4299.286516] Lustre: Skipped 5 previous similar messages [ 4299.313263] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4299.328185] Lustre: Skipped 8 previous similar messages [ 4302.308422] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4305.230951] Lustre: server umount lustre-MDT0000 complete [ 4307.432771] LustreError: 124020:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4307.443599] LustreError: 124020:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 4309.430491] LustreError: 124005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788578857 with bad export cookie 18226594845387791432 [ 4309.438726] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4309.447699] LustreError: 124005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4312.547364] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4312.574920] Lustre: Skipped 2 previous similar messages [ 4312.584920] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4312.597748] Lustre: Skipped 5 previous similar messages [ 4315.996637] Lustre: server umount lustre-MDT0001 complete [ 4325.650236] Lustre: server umount lustre-OST0000 complete [ 4334.870706] Lustre: server umount lustre-OST0001 complete [ 4342.664147] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_hostid [ 4357.466962] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 4395.990833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4396.297783] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4396.370725] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4396.490805] Lustre: lustre-MDT0000: new disk, initializing [ 4396.576451] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4396.582544] Lustre: Skipped 5 previous similar messages [ 4396.602425] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4402.034705] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 4413.610934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4413.756463] Lustre: 135944:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4413.845490] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4413.848584] Lustre: Skipped 1 previous similar message [ 4413.915477] Lustre: lustre-MDT0001: new disk, initializing [ 4414.097591] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4414.104458] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4419.373145] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 4428.295940] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4434.265067] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4441.832409] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4442.138548] Lustre: lustre-OST0000: new disk, initializing [ 4442.147939] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4442.155437] Lustre: 137915:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4442.232226] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4442.240961] Lustre: Skipped 1 previous similar message [ 4443.510152] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4443.515384] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4443.643142] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4449.309493] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 4458.899506] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4468.537666] Lustre: lustre-MDD0000: changelog on [ 4471.535274] Lustre: lustre-MDD0001: changelog on [ 6180.452260] Lustre: lustre-MDD0001: changelog off [ 6182.503221] Lustre: 135950:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6183.470596] Lustre: lustre-MDD0000: changelog off [ 6194.662856] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6194.667166] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6194.678094] Lustre: Skipped 1 previous similar message [ 6194.702366] Lustre: Skipped 3 previous similar messages [ 6195.897957] Lustre: server umount lustre-OST0000 complete [ 6200.812095] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6200.823759] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6200.850164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6204.905824] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6204.908721] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6206.286513] Lustre: server umount lustre-MDT0000 complete [ 6210.025294] LustreError: 139880:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6210.041313] LustreError: 136741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788580757 with bad export cookie 18226594845388608556 [ 6210.041846] LustreError: 139880:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 6210.044895] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6210.053430] LustreError: 136741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6210.430782] Lustre: server umount lustre-MDT0001 complete [ 6223.297404] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 6226.082985] Key type lgssc unregistered [ 6226.346841] LNet: 147023:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6226.350769] LNetError: 147023:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6226.363821] LNet: Removed LNI 192.168.201.133@tcp [ 6227.498169] Key type .llcrypt unregistered [ 6227.501176] Key type ._llcrypt unregistered [ 6249.613889] Key type ._llcrypt registered [ 6249.617419] Key type .llcrypt registered [ 6249.770279] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_hostid [ 6267.417637] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 6268.771235] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6268.819085] alg: No test for adler32 (adler32-zlib) [ 6269.878475] Lustre: Lustre: Build Version: 2.17.58_3_g4a08d7b [ 6270.116235] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 6271.832313] Key type lgssc registered [ 6273.638220] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6310.223443] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6310.247801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6311.515224] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6311.546580] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6311.633462] Lustre: lustre-MDT0000: new disk, initializing [ 6311.759857] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6311.784142] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6316.976863] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 6328.259744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6328.392045] Lustre: 150224:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6328.448961] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6328.455458] Lustre: Skipped 1 previous similar message [ 6328.550370] Lustre: lustre-MDT0001: new disk, initializing [ 6328.654527] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6328.700809] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6328.711708] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6333.915590] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 6341.999718] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6347.182365] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6354.208771] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6354.418415] Lustre: lustre-OST0000: new disk, initializing [ 6354.422481] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6354.429354] Lustre: 152204:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6354.515519] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6361.185235] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6361.208498] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6361.261640] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 6361.375323] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6371.365669] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6378.549563] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_module obdecho/obdecho [ 6380.229958] LustreError: 153206:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6380.243336] LustreError: 153206:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6382.124748] LustreError: 153302:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6395.361253] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6395.374313] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6395.395797] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6397.416093] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6397.436142] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6397.455609] Lustre: Skipped 1 previous similar message [ 6397.481606] Lustre: Skipped 2 previous similar messages [ 6399.763446] Lustre: server umount lustre-MDT0000 complete [ 6404.150581] LustreError: 150214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788580951 with bad export cookie 9290980057905816362 [ 6404.168635] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6404.589723] Lustre: server umount lustre-MDT0001 complete [ 6418.707603] Lustre: server umount lustre-OST0000 complete [ 6442.956172] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_hostid [ 6454.163475] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 6496.534469] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6496.810436] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6496.906121] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6497.064378] Lustre: lustre-MDT0000: new disk, initializing [ 6497.273913] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6497.296673] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6501.645601] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 6513.948754] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6514.156400] Lustre: 157022:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6514.268466] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6514.273402] Lustre: Skipped 1 previous similar message [ 6514.383261] Lustre: lustre-MDT0001: new disk, initializing [ 6514.603799] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6514.667261] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6514.695486] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6522.546265] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 6534.616914] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6540.643154] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6547.967777] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6548.432236] Lustre: lustre-OST0000: new disk, initializing [ 6548.439810] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6548.453902] Lustre: 158993:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6548.630258] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6555.749875] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 6558.292928] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6558.313244] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6558.424289] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6566.374453] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6571.229209] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6571.500615] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6576.952880] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6577.274456] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6579.169995] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6579.174719] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6579.179659] Lustre: Skipped 1 previous similar message [ 6579.183760] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6584.292027] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6584.313569] Lustre: Skipped 2 previous similar messages [ 6585.524955] Lustre: server umount lustre-OST0000 complete [ 6591.459328] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6591.468932] LustreError: Skipped 1 previous similar message [ 6591.479308] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6591.503623] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6596.480534] Lustre: server umount lustre-MDT0000 complete [ 6599.649025] LustreError: 157036:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6599.662960] LustreError: 157036:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 6600.312030] LustreError: 157820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788581147 with bad export cookie 9290980057905817321 [ 6600.315843] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6600.327704] LustreError: 157820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 6600.881257] Lustre: server umount lustre-MDT0001 complete [ 6605.064216] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 00:05:50 (1788581150) [ 6606.872626] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6609.042914] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 00:05:54 (1788581154) [ 6617.934381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6618.647583] LustreError: 161405:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6618.774738] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6624.231071] LustreError: 161406:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6624.703158] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 6628.328678] LustreError: 161405:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6633.442958] LustreError: 161406:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6635.037938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6640.911892] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 6649.990882] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6656.744984] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6664.624237] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6664.985261] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6664.997135] Lustre: Skipped 1 previous similar message [ 6672.868872] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 6681.484960] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6698.982496] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6698.992101] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6699.004543] Lustre: Skipped 2 previous similar messages [ 6699.014876] Lustre: Skipped 4 previous similar messages [ 6705.030766] Lustre: server umount lustre-MDT0000 complete [ 6707.706962] LustreError: 161400:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6709.146868] LustreError: 161385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788581256 with bad export cookie 9290980057905818210 [ 6709.149311] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6709.178148] LustreError: 161385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6709.226583] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6709.251197] Lustre: Skipped 1 previous similar message [ 6715.835256] Lustre: server umount lustre-MDT0001 complete [ 6723.015901] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6737.037227] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6737.364327] LustreError: 165268:0:(llog.c:1655:llog_backup()) MGC192.168.201.133@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6737.384117] Lustre: 165268:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.133@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6738.914836] LustreError: 165273:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6738.922167] LustreError: 165273:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6773.723795] LustreError: 165275:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6773.777523] LustreError: 165275:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 24 previous similar messages [ 6798.304260] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6798.311063] Lustre: 165268:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.133@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6838.755036] LustreError: 165275:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6838.783503] LustreError: 165275:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 44 previous similar messages [ 6839.300219] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6844.397972] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6844.515420] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 6854.379321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6858.735702] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 6884.712889] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 6884.849734] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6892.880698] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6894.515745] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6900.477080] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6901.996590] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6920.675828] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6920.685362] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6920.700562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6920.705874] Lustre: Skipped 1 previous similar message [ 6920.717356] Lustre: Skipped 9 previous similar messages [ 6924.615860] Lustre: server umount lustre-MDT0000 complete [ 6928.638713] LustreError: 161386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788581476 with bad export cookie 9290980057905824678 [ 6928.643351] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6928.649806] LustreError: 161386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6929.015741] Lustre: server umount lustre-MDT0001 complete [ 6943.549706] Lustre: server umount lustre-OST0000 complete [ 6966.071329] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_hostid [ 6979.301090] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 7019.152444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7019.517802] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7019.574350] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7019.705160] Lustre: lustre-MDT0000: new disk, initializing [ 7019.829547] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7019.842679] Lustre: Skipped 1 previous similar message [ 7019.864828] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7025.111624] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 7035.720164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7035.878603] Lustre: 170379:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7036.284578] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7040.561774] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 7049.801516] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7056.723119] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7065.403053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7065.818129] Lustre: lustre-OST0000: new disk, initializing [ 7065.821311] Lustre: Skipped 1 previous similar message [ 7065.825827] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7065.828745] Lustre: Skipped 2 previous similar messages [ 7065.831666] Lustre: 172345:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7072.188166] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 7075.904819] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7075.920048] Lustre: Skipped 1 previous similar message [ 7075.927643] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7076.085584] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7081.875453] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7087.200238] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7087.463969] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7092.616742] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7092.783923] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7096.821611] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7096.831176] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7096.831795] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7096.847151] Lustre: Skipped 2 previous similar messages [ 7100.591750] Lustre: server umount lustre-OST0000 complete [ 7111.764199] Lustre: server umount lustre-MDT0000 complete [ 7112.161617] LustreError: 170389:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7112.182578] LustreError: 170389:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 40 previous similar messages [ 7117.245221] LustreError: 170370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788581664 with bad export cookie 9290980057905841492 [ 7117.247920] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7117.251023] LustreError: 170370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7117.667749] Lustre: server umount lustre-MDT0001 complete [ 7127.742281] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 00:14:33 (1788581673) [ 7134.551909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7138.509899] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 7148.554276] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7155.258572] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 7164.207975] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7171.975606] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7180.004454] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7187.583953] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 7195.794716] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7216.482608] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 00:16:01 (1788581761) [ 7224.807430] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7224.812456] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7224.835923] Lustre: Skipped 2 previous similar messages [ 7224.857659] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7224.863837] Lustre: Skipped 3 previous similar messages [ 7227.461141] Lustre: server umount lustre-OST0000 complete [ 7237.779735] Lustre: server umount lustre-MDT0000 complete [ 7243.413324] LustreError: 177760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788581791 with bad export cookie 9290980057905842374 [ 7243.413828] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7243.424071] LustreError: 177760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7243.999142] Lustre: server umount lustre-MDT0001 complete [ 7257.454589] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 7261.630595] Key type lgssc unregistered [ 7261.911899] LNet: 178511:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7261.918672] LNetError: 178511:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7262.950404] LNet: Removed LNI 192.168.201.133@tcp [ 7264.286854] Key type .llcrypt unregistered [ 7264.292967] Key type ._llcrypt unregistered [ 7284.471091] Key type ._llcrypt registered [ 7284.476582] Key type .llcrypt registered [ 7284.639417] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 7285.709609] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7285.742563] alg: No test for adler32 (adler32-zlib) [ 7287.186625] Lustre: Lustre: Build Version: 2.17.58_3_g4a08d7b [ 7287.582741] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 7289.352288] Key type lgssc registered [ 7291.025698] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7300.007721] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7300.039478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7315.680406] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7320.800337] LustreError: 179749:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.133@tcp: failed processing log, type 4: rc = -110 [ 7325.920632] LustreError: 179749:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7325.933299] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7325.947719] LustreError: MGC192.168.201.133@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7325.968617] LustreError: 179749:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7325.980199] LustreError: 179749:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -22 [ 7325.986854] LustreError: 179749:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 7326.055256] Lustre: server umount lustre-OST0000 complete [ 7326.067224] LustreError: 179749:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7335.077228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7336.489796] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7341.293459] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 7346.663560] LustreError: 180249:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7346.693949] LustreError: 180249:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7350.155164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7350.467849] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7355.021589] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 7362.469947] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7368.318304] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7374.877338] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7375.063993] LustreError: 182064:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7375.074928] Lustre: 182064:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.201.133@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7375.227800] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7381.380279] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 7388.994213] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7391.730415] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7391.732244] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7391.742094] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7391.753331] Lustre: Skipped 1 previous similar message [ 7391.760899] Lustre: Skipped 1 previous similar message [ 7396.840865] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7396.853139] Lustre: Skipped 2 previous similar messages [ 7397.427797] Lustre: server umount lustre-MDT0000 complete [ 7400.951386] LustreError: 182824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788581948 with bad export cookie 7240968175627040523 [ 7400.957875] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7400.961494] LustreError: 182824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7401.279905] Lustre: server umount lustre-MDT0001 complete [ 7415.389882] Lustre: server umount lustre-OST0000 complete [ 7425.730718] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_hostid [ 7436.090247] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 7470.602781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7470.894626] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7470.955388] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7471.071346] Lustre: lustre-MDT0000: new disk, initializing [ 7471.200729] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7471.223581] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7475.752943] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 7487.682234] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7487.895427] Lustre: 185591:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7487.980994] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7487.988531] Lustre: Skipped 1 previous similar message [ 7488.108761] Lustre: lustre-MDT0001: new disk, initializing [ 7488.244303] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7488.282083] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7488.303124] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7493.427940] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 7503.071298] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7508.670387] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7516.291071] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7516.606831] Lustre: lustre-OST0000: new disk, initializing [ 7516.611157] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7516.619252] Lustre: 187562:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7516.707621] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7523.810859] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 7524.450754] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7524.458661] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7524.546293] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7533.682480] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7539.419732] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7539.668725] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7543.937082] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7544.102291] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7550.438189] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7550.440921] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7550.470985] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7550.471185] Lustre: Skipped 2 previous similar messages [ 7550.479037] Lustre: Skipped 1 previous similar message [ 7555.575638] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7555.590718] Lustre: Skipped 1 previous similar message [ 7560.161772] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7560.714413] Lustre: server umount lustre-OST0000 complete [ 7565.282063] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7565.293041] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7565.314695] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7571.503143] Lustre: server umount lustre-MDT0000 complete [ 7575.549058] LustreError: 185583:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788582123 with bad export cookie 7240968175627041251 [ 7575.559199] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7575.562827] LustreError: 185583:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7575.948892] Lustre: server umount lustre-MDT0001 complete [ 7585.587893] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 00:22:11 (1788582131) [ 7604.755641] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 7607.908269] Key type lgssc unregistered [ 7608.178186] LNet: 190505:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7608.190659] LNetError: 190505:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7609.257777] LNet: Removed LNI 192.168.201.133@tcp [ 7610.355208] Key type .llcrypt unregistered [ 7610.357423] Key type ._llcrypt unregistered [ 7629.815123] Key type ._llcrypt registered [ 7629.818630] Key type .llcrypt registered [ 7629.922940] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 7631.312934] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7631.424169] alg: No test for adler32 (adler32-zlib) [ 7632.559215] Lustre: Lustre: Build Version: 2.17.58_3_g4a08d7b [ 7632.860812] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 7634.584344] Key type lgssc registered [ 7635.604247] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7643.719412] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7643.737220] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7645.288678] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7650.408463] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 7654.891294] LustreError: 191721:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7654.919966] LustreError: 191721:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7658.743705] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7659.176261] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7664.231532] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 7671.936172] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7676.765807] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7684.641334] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7684.925918] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7684.928700] LustreError: 193584:0:(llog.c:1696:llog_backup()) MGC192.168.201.133@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7684.937289] Lustre: 193584:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.201.133@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7685.094848] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7691.716104] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 7699.832988] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7703.343157] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7703.351341] Lustre: Skipped 1 previous similar message [ 7703.353122] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7703.364549] Lustre: Skipped 2 previous similar messages [ 7708.651055] LustreError: 191722:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7708.678818] LustreError: 191722:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7708.830414] Lustre: server umount lustre-MDT0000 complete [ 7713.345180] LustreError: 191701:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788582261 with bad export cookie 12130793016590579623 [ 7713.353435] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7713.363175] LustreError: 191701:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7713.592238] LustreError: 192451:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000bb7d74d8 x1875464348003712/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7713.730657] Lustre: server umount lustre-MDT0001 complete [ 7727.795142] Lustre: server umount lustre-OST0000 complete [ 7739.956504] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_hostid [ 7754.346807] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 7790.870830] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7791.103632] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7791.146654] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7791.248761] Lustre: lustre-MDT0000: new disk, initializing [ 7791.319549] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7791.335234] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7794.904534] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 7805.132561] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7805.286915] Lustre: 197122:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7805.328918] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7805.334769] Lustre: Skipped 1 previous similar message [ 7805.336108] LustreError: 197907:0:(llog.c:1696:llog_backup()) MGC192.168.201.133@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7805.340957] LustreError: 197907:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7805.345293] Lustre: 197907:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.133@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7805.355549] LustreError: 197133:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7805.378817] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7805.385361] Lustre: Skipped 1 previous similar message [ 7805.506974] Lustre: lustre-MDT0001: new disk, initializing [ 7805.706942] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7805.792819] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7805.817452] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7811.489803] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 7821.458479] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7828.376352] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7835.532939] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7835.777975] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7835.780627] Lustre: Skipped 1 previous similar message [ 7835.787455] LustreError: 199068:0:(llog.c:1696:llog_backup()) MGC192.168.201.133@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7835.796336] LustreError: 199068:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7835.803820] Lustre: 199068:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.133@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7835.815281] Lustre: 199068:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7835.844151] Lustre: lustre-OST0000: new disk, initializing [ 7835.852299] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7835.864949] Lustre: 199091:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7836.007430] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7842.441875] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 7845.499359] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7845.506440] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7845.702170] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7852.021222] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7856.783771] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7856.977703] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7862.161793] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7862.368306] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7866.343891] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7866.344975] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7866.352563] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7866.375599] Lustre: Skipped 2 previous similar messages [ 7870.545507] Lustre: server umount lustre-OST0000 complete [ 7876.583057] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7876.586678] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7876.594807] Lustre: Skipped 1 previous similar message [ 7876.597104] Lustre: Skipped 1 previous similar message [ 7880.568113] Lustre: server umount lustre-MDT0000 complete [ 7881.697344] LustreError: 198145:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7881.716725] LustreError: 198145:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7884.420743] LustreError: 197919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788582432 with bad export cookie 12130793016590580344 [ 7884.425365] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7884.431799] LustreError: 197919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7884.871857] Lustre: server umount lustre-MDT0001 complete [ 7894.491984] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 00:27:19 (1788582439) [ 7909.531952] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 7918.915996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7919.268500] LustreError: 202218:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7919.325127] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7924.244685] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 7928.817397] LustreError: 202219:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7928.842985] LustreError: 202219:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7933.653969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7940.123786] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 7943.897951] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7951.172902] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7951.545551] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7951.556958] Lustre: Skipped 1 previous similar message [ 7957.825956] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 7965.538985] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7965.667151] Lustre: lustre-OST0001: new disk, initializing [ 7965.672636] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7965.680968] Lustre: 204524:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7971.802752] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 7973.557470] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7973.572610] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7973.917932] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7986.339815] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7994.250092] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7996.198754] Lustre: DEBUG MARKER: ADD OST3 [ 8001.016643] loop: module loaded [ 8001.122679] loop0: detected capacity change from 0 to 204800000 [ 8001.444977] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8001.463848] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8001.482586] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8007.703280] loop0: detected capacity change from 0 to 204800000 [ 8008.715958] Lustre: DEBUG MARKER: STOP OST3 [ 8014.209817] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8014.458631] Lustre: lustre-OST0003: new disk, initializing [ 8014.465845] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8014.471582] Lustre: 206532:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8016.242646] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8016.356053] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8016.368518] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8016.421764] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8018.272075] LustreError: 206696:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8018.518391] Lustre: Failing over lustre-OST0003 [ 8018.654588] Lustre: server umount lustre-OST0003 complete [ 8021.476726] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8021.484340] LustreError: Skipped 1 previous similar message [ 8021.493973] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8021.510966] Lustre: Skipped 1 previous similar message [ 8021.521105] LustreError: 203700:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8021.536610] LustreError: 203700:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8022.152692] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 8031.138386] Lustre: DEBUG MARKER: START OST3 again [ 8038.583429] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8038.920508] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8038.930250] Lustre: Skipped 2 previous similar messages [ 8038.952004] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8040.061774] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8040.435558] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8040.439118] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8046.223712] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 8069.445560] Lustre: Failing over lustre-OST0003 [ 8069.560678] Lustre: server umount lustre-OST0003 complete [ 8071.142611] LustreError: lustre-OST0003-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8071.156708] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8071.157134] LustreError: Skipped 1 previous similar message [ 8071.157914] LustreError: 203700:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8071.157921] LustreError: 203700:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 8071.218923] Lustre: Skipped 1 previous similar message [ 8092.641789] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8092.665071] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8092.702403] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8092.705205] Lustre: Skipped 1 previous similar message [ 8097.962422] Lustre: server umount lustre-MDT0000 complete [ 8101.400083] LustreError: 202198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788582649 with bad export cookie 12130793016590581226 [ 8101.406896] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8101.419074] LustreError: 202198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8103.400197] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8103.406859] Lustre: Skipped 5 previous similar messages [ 8108.000863] Lustre: server umount lustre-MDT0001 complete [ 8112.649654] Lustre: server umount lustre-OST0000 complete [ 8117.243658] Lustre: server umount lustre-OST0001 complete [ 8125.876434] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_hostid [ 8136.035336] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 8176.736460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8177.112860] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8177.174832] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8177.340768] Lustre: lustre-MDT0000: new disk, initializing [ 8177.435914] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8177.468877] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8182.505050] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 8194.918664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8195.052538] Lustre: 211985:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8195.111451] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8195.115437] Lustre: Skipped 1 previous similar message [ 8195.171354] Lustre: lustre-MDT0001: new disk, initializing [ 8195.317846] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8195.333913] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8199.346731] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 8209.963822] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8217.745653] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8225.645319] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8226.024794] Lustre: 213956:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8227.235252] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8227.379879] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8233.541645] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 8242.703637] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8250.177761] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8250.474567] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8257.430512] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8257.713823] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8263.138774] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8263.145406] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8263.154394] Lustre: Skipped 5 previous similar messages [ 8263.163736] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8263.173230] Lustre: Skipped 3 previous similar messages [ 8266.462392] Lustre: server umount lustre-OST0000 complete [ 8272.354055] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8276.831200] Lustre: server umount lustre-MDT0000 complete [ 8278.498149] LustreError: 211997:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8278.531392] LustreError: 211997:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 20 previous similar messages [ 8280.858079] LustreError: 211979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788582828 with bad export cookie 12130793016590583522 [ 8280.870190] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8280.882331] LustreError: 211979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8281.196690] Lustre: server umount lustre-MDT0001 complete [ 8284.960946] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 00:33:50 (1788582830) [ 8302.461474] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_hostid [ 8313.882108] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 8352.979106] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8353.261785] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8353.288289] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8353.294371] Lustre: Skipped 1 previous similar message [ 8353.402734] Lustre: lustre-MDT0000: new disk, initializing [ 8353.404717] Lustre: Skipped 1 previous similar message [ 8353.524347] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8353.535278] Lustre: Skipped 1 previous similar message [ 8359.102327] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 8370.609035] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8370.758785] Lustre: 218567:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8371.066988] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8376.047868] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 8386.655928] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8392.711584] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8400.034527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8400.316189] Lustre: 220536:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8402.047521] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8406.360769] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 8415.754026] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8420.401934] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8420.573320] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8426.438342] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8426.755805] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8432.611879] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8432.622298] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8432.633979] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8432.647686] Lustre: Skipped 4 previous similar messages [ 8432.666200] Lustre: Skipped 4 previous similar messages [ 8443.361588] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8443.526698] Lustre: server umount lustre-OST0000 complete [ 8454.505698] Lustre: server umount lustre-MDT0000 complete [ 8457.414690] LustreError: 218557:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788583005 with bad export cookie 12130793016590584404 [ 8457.426819] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8457.762368] Lustre: server umount lustre-MDT0001 complete [ 8464.672178] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8465.066719] LustreError: 222758:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8465.135965] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8465.145575] Lustre: Skipped 5 previous similar messages [ 8469.215128] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 8477.830488] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8482.683289] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 8489.775449] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8495.660588] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8501.543057] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8508.693957] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 8517.274149] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8528.867559] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8528.884515] LustreError: Skipped 1 previous similar message [ 8528.892542] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8528.904371] Lustre: Skipped 2 previous similar messages [ 8528.912787] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8528.925431] Lustre: Skipped 6 previous similar messages [ 8533.820497] Lustre: server umount lustre-OST0000 complete [ 8544.509472] Lustre: server umount lustre-MDT0000 complete [ 8548.708310] LustreError: 222738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788583096 with bad export cookie 12130793016590585286 [ 8548.723209] LustreError: 222738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8548.727648] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8549.035264] Lustre: server umount lustre-MDT0001 complete [ 8656.746714] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 8659.688780] Key type lgssc unregistered [ 8660.018523] LNet: 226418:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8660.025959] LNetError: 226418:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8660.038982] LNet: Removed LNI 192.168.201.133@tcp [ 8660.843176] Key type .llcrypt unregistered [ 8660.844942] Key type ._llcrypt unregistered [ 8687.052857] Key type ._llcrypt registered [ 8687.055464] Key type .llcrypt registered [ 8687.204579] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_hostid [ 8704.682495] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 8705.436202] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8705.593540] alg: No test for adler32 (adler32-zlib) [ 8706.774674] Lustre: Lustre: Build Version: 2.17.58_3_g4a08d7b [ 8706.918247] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 8708.568171] Key type lgssc registered [ 8710.161224] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8743.717316] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8743.764067] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8745.000851] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8745.055684] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8745.168769] Lustre: lustre-MDT0000: new disk, initializing [ 8745.235306] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8745.248644] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8750.004420] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 8761.724352] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8761.879727] Lustre: 229770:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8761.923408] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8761.927023] Lustre: Skipped 1 previous similar message [ 8762.012626] Lustre: lustre-MDT0001: new disk, initializing [ 8762.124255] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8762.165106] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8762.182103] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8767.450562] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 8777.156867] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8783.106534] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8789.665974] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8789.890632] Lustre: lustre-OST0000: new disk, initializing [ 8789.894968] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8789.902668] Lustre: 231739:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8789.963701] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8795.719639] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 8796.193136] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8796.200350] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8796.279447] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8805.424887] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8810.466053] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8810.790439] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8816.346669] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8816.555934] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8821.732971] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8821.737969] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8821.738567] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8821.763109] Lustre: Skipped 1 previous similar message [ 8824.916931] Lustre: server umount lustre-OST0000 complete [ 8828.898139] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8828.917335] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8828.930431] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8828.936212] Lustre: Skipped 1 previous similar message [ 8831.970509] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8831.973113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8832.052546] Lustre: Skipped 1 previous similar message [ 8837.095419] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8837.105834] Lustre: Skipped 1 previous similar message [ 8842.221322] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8842.234223] Lustre: Skipped 1 previous similar message [ 8843.232778] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8843.519171] Lustre: server umount lustre-MDT0000 complete [ 8847.662444] LustreError: 229760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788583395 with bad export cookie 10573267995474894427 [ 8847.677370] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8847.687164] LustreError: 229760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8848.053260] Lustre: server umount lustre-MDT0001 complete [ 8870.078311] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 8882.711037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8883.412753] LustreError: 234675:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8883.445703] LustreError: 234675:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8883.511972] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8888.662521] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 8888.802878] LustreError: 234676:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8893.923962] LustreError: 234675:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8898.253787] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8898.719272] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8903.081662] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 8906.229800] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8913.322093] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8913.751497] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8919.135693] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 8927.853485] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8928.143505] Lustre: lustre-OST0001: new disk, initializing [ 8928.153119] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8928.168335] Lustre: 236983:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8936.587285] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8936.600696] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8936.731129] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8936.785561] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 8949.608530] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8955.544857] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8968.052840] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 00:45:13 (1788583513) [ 8970.210591] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8970.216861] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8970.231867] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8976.413990] Lustre: server umount lustre-MDT0000 complete [ 8977.895202] LustreError: 234670:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8977.915638] LustreError: 234670:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8980.239671] LustreError: 235411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788583527 with bad export cookie 10573267995474895309 [ 8980.261220] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8980.662463] Lustre: server umount lustre-MDT0001 complete [ 8994.932471] Lustre: server umount lustre-OST0000 complete [ 8998.369303] Lustre: 227995:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788583530/real 1788583530] req@ffff9852ecd9ce00 x1875465474258560/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788583546 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8998.405948] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8998.423698] Lustre: Skipped 3 previous similar messages [ 8999.689886] Lustre: server umount lustre-OST0001 complete [ 9007.854693] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_hostid [ 9019.649767] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 9060.710522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9061.148284] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9061.185050] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9061.323577] Lustre: lustre-MDT0000: new disk, initializing [ 9061.440627] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9061.451594] Lustre: Skipped 1 previous similar message [ 9061.475792] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9067.474529] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 9081.067475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9081.225168] Lustre: 240928:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9081.276312] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9081.279728] Lustre: Skipped 1 previous similar message [ 9081.352290] Lustre: lustre-MDT0001: new disk, initializing [ 9081.536146] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9081.557204] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9087.645334] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 9098.559681] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9104.866956] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9111.911779] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9112.201351] Lustre: lustre-OST0000: new disk, initializing [ 9112.203690] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9112.211142] Lustre: 242901:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9112.340417] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9112.350747] Lustre: Skipped 1 previous similar message [ 9114.215698] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9114.224977] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9114.302922] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9118.336239] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 9129.265510] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9136.234239] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9136.578977] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9143.646485] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9144.109299] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9150.435256] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9150.455380] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9150.485290] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9150.495526] Lustre: Skipped 5 previous similar messages [ 9153.502350] Lustre: server umount lustre-OST0000 complete [ 9158.113024] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9158.115057] LustreError: Skipped 1 previous similar message [ 9164.061832] Lustre: server umount lustre-MDT0000 complete [ 9165.795225] LustreError: 241771:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9165.832787] LustreError: 241771:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9168.553065] LustreError: 241725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788583716 with bad export cookie 10573267995474897444 [ 9168.554995] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9168.564534] LustreError: 241725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 9169.056987] Lustre: server umount lustre-MDT0001 complete [ 9182.369970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9182.725993] LustreError: 245120:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9182.834808] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9187.231789] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 9196.926716] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9202.125435] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 9210.070435] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9214.940405] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9221.471323] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9228.055211] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 9235.546141] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9381.862614] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9381.884987] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9381.885799] LustreError: Skipped 1 previous similar message [ 9381.891042] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9381.898728] Lustre: Skipped 4 previous similar messages [ 9381.919403] Lustre: Skipped 4 previous similar messages [ 9386.984802] Lustre: server umount lustre-OST0000 complete [ 9396.581254] Lustre: server umount lustre-MDT0000 complete [ 9397.231623] LustreError: 245121:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9397.251373] LustreError: 245121:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9401.023107] LustreError: 245101:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788583948 with bad export cookie 10573267995474898326 [ 9401.049759] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9401.630928] Lustre: server umount lustre-MDT0001 complete [ 9505.820668] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 9509.031554] Key type lgssc unregistered [ 9509.458142] LNet: 248729:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9509.466894] LNetError: 248729:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9509.493880] LNet: Removed LNI 192.168.201.133@tcp [ 9510.431516] Key type .llcrypt unregistered [ 9510.436363] Key type ._llcrypt unregistered [ 9537.562427] Key type ._llcrypt registered [ 9537.568266] Key type .llcrypt registered [ 9537.731441] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_hostid [ 9556.478404] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 9557.515667] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9557.622472] alg: No test for adler32 (adler32-zlib) [ 9558.829172] Lustre: Lustre: Build Version: 2.17.58_3_g4a08d7b [ 9559.153607] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 9560.888176] Key type lgssc registered [ 9562.518534] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9613.217829] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 9625.329134] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9625.380238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9626.787522] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9626.829714] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9626.974207] Lustre: lustre-MDT0000: new disk, initializing [ 9627.115341] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9627.139809] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9632.963624] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 9644.506517] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9644.613756] Lustre: 252794:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9644.695272] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9644.707165] Lustre: Skipped 1 previous similar message [ 9644.801607] Lustre: lustre-MDT0001: new disk, initializing [ 9644.900539] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9644.938135] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9644.950242] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9650.861367] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 9656.419820] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9663.820518] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9664.240594] Lustre: lustre-OST0000: new disk, initializing [ 9664.246335] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9664.255888] Lustre: 254425:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9664.371533] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9667.910808] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9667.922304] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9667.985564] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9671.482424] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 9682.688558] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9682.869518] Lustre: lustre-OST0001: new disk, initializing [ 9682.873785] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9682.879730] Lustre: 255297:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9682.986249] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9688.728209] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9688.743845] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9688.912657] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9689.442337] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 9700.530911] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9713.579786] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9726.946052] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9726.956200] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9726.968106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9730.037651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9730.043546] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9730.087034] Lustre: Skipped 1 previous similar message [ 9733.230742] Lustre: server umount lustre-MDT0000 complete [ 9737.901432] LustreError: 255296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788584285 with bad export cookie 7492354374345631194 [ 9737.905183] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9737.917250] LustreError: 255296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9738.564649] Lustre: server umount lustre-MDT0001 complete [ 9753.869988] Lustre: server umount lustre-OST0000 complete [ 9755.552154] Lustre: 250300:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788584287/real 1788584287] req@ffff9851c7085c00 x1875466367613056/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788584303 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9755.583898] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9755.595900] Lustre: Skipped 1 previous similar message [ 9757.871974] Lustre: server umount lustre-OST0001 complete [ 9773.648407] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9785.545079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9795.450328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9806.651806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9829.282768] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 9839.721256] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9839.754279] Lustre: lustre-MDT0000: reset Object Index mappings [ 9840.409804] LustreError: 260313:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9840.439363] LustreError: 260313:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 9840.542094] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9845.732048] LustreError: 260314:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9845.934177] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 9850.850388] LustreError: 260313:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9855.975920] LustreError: 260314:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9857.086463] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9857.103773] Lustre: lustre-MDT0001: reset Object Index mappings [ 9857.543637] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9862.701870] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 9866.234928] Lustre: 261455:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9874.523237] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9875.065468] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9882.866919] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 9883.246067] LustreError: 261811:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9886.363754] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9891.834363] LustreError: 261812:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9891.861831] LustreError: 261812:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9893.940217] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9899.519607] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9903.232359] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 9912.890686] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9917.322473] Lustre: 263340:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9925.099402] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9925.106742] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9925.116299] Lustre: Skipped 3 previous similar messages [ 9925.132326] Lustre: Skipped 6 previous similar messages [ 9930.219174] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9930.230496] Lustre: Skipped 3 previous similar messages [ 9930.596830] Lustre: server umount lustre-MDT0000 complete [ 9934.910368] LustreError: 260294:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788584482 with bad export cookie 7492354374345637984 [ 9934.931207] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9935.335458] LustreError: 261832:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9935.355700] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9935.361202] LustreError: 261832:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9935.658676] Lustre: server umount lustre-MDT0001 complete [ 9941.490369] Lustre: server umount lustre-OST0000 complete [ 9946.119916] Lustre: server umount lustre-OST0001 complete [ 9972.253141] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 01:01:57 (1788584517) [ 9992.358512] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_hostid [10004.537651] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [10045.467670] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10045.782178] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10045.821256] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10045.963717] Lustre: lustre-MDT0000: new disk, initializing [10046.088788] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10046.091372] Lustre: Skipped 1 previous similar message [10046.135811] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10052.240469] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [10065.821274] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10066.013812] Lustre: 267598:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10066.091326] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10066.097322] Lustre: Skipped 1 previous similar message [10066.241457] Lustre: lustre-MDT0001: new disk, initializing [10066.446829] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10066.456794] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10072.356238] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [10082.794833] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10089.488545] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10097.267598] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10097.532590] Lustre: lustre-OST0000: new disk, initializing [10097.535695] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10097.540462] Lustre: 269565:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10099.005689] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10099.023086] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10099.151233] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10104.969618] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [10116.504535] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10122.375371] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10122.547300] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10129.530558] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10129.901831] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10135.010520] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10135.013247] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10135.028628] LustreError: Skipped 1 previous similar message [10135.029457] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10135.060521] Lustre: Skipped 2 previous similar messages [10138.662229] Lustre: server umount lustre-OST0000 complete [10145.257051] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10145.265487] Lustre: Skipped 1 previous similar message [10149.953784] Lustre: server umount lustre-MDT0000 complete [10150.370874] LustreError: 267603:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10150.380085] LustreError: 267603:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10154.274691] LustreError: 267590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788584701 with bad export cookie 7492354374345640077 [10154.279061] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10154.289199] LustreError: 267590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10154.615224] Lustre: server umount lustre-MDT0001 complete [10178.365363] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [10190.144741] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10190.800080] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10190.817523] Lustre: Skipped 2 previous similar messages [10195.363978] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [10204.340008] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10210.805621] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [10214.115367] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10222.141882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10230.582918] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [10241.565774] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10241.808673] Lustre: lustre-OST0001: new disk, initializing [10241.812552] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10241.819070] Lustre: 274814:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10243.970746] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10243.985106] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10244.201987] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10249.927198] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [10260.377961] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10265.047545] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10271.912935] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10271.950888] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10285.029242] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10285.048781] Lustre: Skipped 2 previous similar messages [10285.050922] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10285.062244] Lustre: Skipped 2 previous similar messages [10298.336128] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10298.833250] Lustre: server umount lustre-MDT0000 complete [10300.423600] LustreError: 272500:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10300.460076] LustreError: 272500:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [10303.871212] LustreError: 272486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788584851 with bad export cookie 7492354374345640959 [10303.884114] LustreError: 272486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10303.887843] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10304.495985] Lustre: server umount lustre-MDT0001 complete [10319.774638] Lustre: server umount lustre-OST0000 complete [10321.390809] Lustre: 250302:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788584853/real 1788584853] req@ffff9851cefbea00 x1875466367791232/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788584869 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10321.439623] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10321.464974] Lustre: Skipped 1 previous similar message [10324.676944] Lustre: server umount lustre-OST0001 complete [10341.750615] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [10354.277995] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10354.621098] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10354.623599] Lustre: Skipped 3 previous similar messages [10359.736153] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [10372.877611] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10380.735787] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [10384.314829] Lustre: 279299:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10392.405212] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10399.941138] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10401.220265] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [10411.627932] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10417.140094] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10420.277231] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [10429.035948] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10432.845880] Lustre: 281179:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10438.037364] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10438.198390] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10454.354482] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 01:09:59 (1788584999) [10473.447040] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10473.464728] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10473.474613] Lustre: Skipped 3 previous similar messages [10473.484221] Lustre: Skipped 10 previous similar messages [10477.718565] Lustre: server umount lustre-MDT0000 complete [10478.575462] LustreError: 280994:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10478.606805] LustreError: 280994:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 14 previous similar messages [10481.779653] LustreError: 278140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788585029 with bad export cookie 7492354374345647805 [10481.786328] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10488.079854] Lustre: server umount lustre-MDT0001 complete [10498.552209] Lustre: server umount lustre-OST0000 complete [10505.096754] Lustre: server umount lustre-OST0001 complete [10516.781508] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_hostid [10526.718099] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [10580.305795] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [10593.173359] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10593.417341] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10593.498490] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10593.595656] Lustre: lustre-MDT0000: new disk, initializing [10593.689982] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10598.953174] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [10610.775414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10610.900117] Lustre: 285712:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10611.207441] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10611.219582] Lustre: Skipped 4 previous similar messages [10611.280615] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10616.397508] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [10629.618680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10630.085026] Lustre: 287344:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10631.892224] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10638.107620] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [10649.881101] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10650.098651] Lustre: 288217:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10651.228710] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10651.242881] Lustre: Skipped 1 previous similar message [10651.381166] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10657.668833] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [10667.633652] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10672.112672] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10672.116590] Lustre: Skipped 1 previous similar message [10688.143447] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 01:13:53 (1788585233) [10692.593872] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10692.602503] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10692.619858] Lustre: Skipped 3 previous similar messages [10692.623057] Lustre: Skipped 5 previous similar messages [10692.813172] Lustre: server umount lustre-MDT0000 complete [10696.285622] LustreError: 285705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788585243 with bad export cookie 7492354374345655288 [10696.285848] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10696.300666] LustreError: 285705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [10696.578818] Lustre: server umount lustre-MDT0001 complete [10709.470888] Lustre: server umount lustre-OST0000 complete [10714.084948] Lustre: 250302:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788585245/real 1788585245] req@ffff9851cf1ea300 x1875466367950848/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788585261 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10714.700746] Lustre: server umount lustre-OST0001 complete [10725.259324] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10731.384405] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [10735.595794] LustreError: 290465:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10735.609932] LustreError: 290465:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [10741.354764] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10747.692595] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [10756.048920] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10762.827054] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10770.515164] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10778.566781] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [10787.045544] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10804.934167] loop1: detected capacity change from 0 to 204800000 [10805.347863] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10805.368040] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10805.401464] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10811.069822] loop1: detected capacity change from 0 to 204800000 [10817.967370] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10818.172573] LustreError: 290453:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs28284: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [10818.190154] LustreError: 290453:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs28284 [10818.200357] LustreError: 290453:0:(mgs_handler.c:517:mgs_target_reg()) fs28284-MDT0001: Error checking target index: rc = -13 [10818.211644] LustreError: fs28284-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10818.230636] LustreError: 294183:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -13 [10818.243891] LustreError: 294183:0:(tgt_mount.c:2132:server_put_super()) no obd fs28284-MDT0001 [10818.253413] LustreError: 294183:0:(tgt_mount.c:116:server_deregister_mount()) fs28284-MDT0001 not registered [10818.285332] Lustre: server umount fs28284-MDT0001 complete [10818.287953] LustreError: 294183:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10828.851270] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10829.099258] Lustre: Setting parameter fs28284-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs28284-MDT0001 [10829.267766] Lustre: srv-fs28284-MDT0001: No data found on store. Initialize space. [10829.284670] Lustre: Skipped 4 previous similar messages [10829.400734] Lustre: fs28284-MDT0001: new disk, initializing [10829.408621] Lustre: Skipped 3 previous similar messages [10829.604666] Lustre: srv-fs28284-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10829.629936] Lustre: 294651:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs28284-MDT0001-mdtlov: can't create new llog: rc = -115 [10829.651540] LustreError: 294651:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs28284-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10834.672797] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [10841.532842] Lustre: server umount fs28284-MDT0001 complete [10853.871299] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10861.063148] loop1: detected capacity change from 0 to 204800000 [10869.201614] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10869.229287] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10869.545864] Lustre: 296219:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs28284-MDT0001-mdtlov: can't create new llog: rc = -115 [10869.552502] LustreError: 296219:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs28284-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10873.999921] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [10879.955546] Lustre: server umount fs28284-MDT0001 complete [10894.822271] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10894.825516] Lustre: Skipped 1 previous similar message [10897.893736] Lustre: server umount lustre-MDT0000 complete [10908.707280] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10908.924545] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10914.823133] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10915.731966] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [10923.879751] loop1: detected capacity change from 0 to 204800000 [10929.953824] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10930.297617] Lustre: 298667:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs28284-MDT0001-mdtlov: can't create new llog: rc = -115 [10930.304163] LustreError: 298667:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs28284-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10935.856431] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [10941.871440] Lustre: server umount fs28284-MDT0001 complete [10971.113792] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10971.134530] Lustre: Skipped 7 previous similar messages [10972.860440] Lustre: server umount lustre-OST0000 complete [10981.345279] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10982.850703] Lustre: server umount lustre-MDT0000 complete [10987.054166] LustreError: 290444:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788585534 with bad export cookie 7492354374345657976 [10987.061121] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10987.069790] LustreError: 290444:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10987.555567] Lustre: server umount lustre-MDT0001 complete [11003.632761] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [11007.600868] Key type lgssc unregistered [11008.009730] LNet: 301011:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11008.035843] LNetError: 301011:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11008.078730] LNet: Removed LNI 192.168.201.133@tcp [11009.275730] Key type .llcrypt unregistered [11009.283555] Key type ._llcrypt unregistered [11038.775495] Key type ._llcrypt registered [11038.777493] Key type .llcrypt registered [11038.900614] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [11040.341844] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11040.410789] alg: No test for adler32 (adler32-zlib) [11041.573702] Lustre: Lustre: Build Version: 2.17.58_3_g4a08d7b [11042.024242] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [11043.818229] Key type lgssc registered [11045.608858] Lustre: Echo OBD driver; http://www.lustre.org/ [11054.148822] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11054.180044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11055.650271] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11060.745197] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [11065.830226] LustreError: 302287:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11065.860150] LustreError: 302287:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11070.946724] LustreError: 302286:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11071.791682] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11072.295239] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11077.692329] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [11087.216426] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11094.369543] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11101.829163] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11102.181936] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11111.656329] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [11116.536550] LustreError: 304115:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11121.318711] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11121.635959] LustreError: 304115:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11125.744648] LustreError: 304116:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11125.794412] LustreError: 304116:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11135.971562] LustreError: 304648:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11136.027221] LustreError: 304648:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [11143.911506] loop1: detected capacity change from 0 to 204800000 [11144.142856] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11144.161160] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11144.183641] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11150.029955] loop1: detected capacity change from 0 to 204800000 [11152.894519] LustreError: 304116:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11152.930252] LustreError: 304116:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [11156.004025] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11156.177424] Lustre: fs1259-OST0000: new disk, initializing [11156.181198] Lustre: srv-fs1259-OST0000: No data found on store. Initialize space. [11156.185562] Lustre: 306024:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11156.254903] Lustre: fs1259-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11162.508712] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [11170.562844] Lustre: server umount fs1259-OST0000 complete [11181.855855] loop1: detected capacity change from 0 to 204800000 [11187.171287] LustreError: 304115:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11187.187536] LustreError: 304115:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 19 previous similar messages [11187.708755] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11187.990277] Lustre: fs1259-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11195.548291] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [11197.418845] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11202.550107] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11204.007660] loop2: detected capacity change from 0 to 204800000 [11204.267919] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11204.300587] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11204.339770] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11209.984059] loop2: detected capacity change from 0 to 204800000 [11216.025244] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11216.162911] LustreError: fs1259-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11216.172894] LustreError: fs1259-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11216.182708] LustreError: 309063:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -13 [11216.192586] LustreError: 309063:0:(tgt_mount.c:2132:server_put_super()) no obd fs1259-OST0001 [11216.200498] LustreError: 309063:0:(tgt_mount.c:116:server_deregister_mount()) fs1259-OST0001 not registered [11216.251046] Lustre: server umount fs1259-OST0001 complete [11216.256375] LustreError: 309063:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11226.998949] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11227.228020] Lustre: fs1259-OST0001: new disk, initializing [11227.232749] Lustre: srv-fs1259-OST0001: No data found on store. Initialize space. [11227.240137] Lustre: 309613:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11227.327377] Lustre: fs1259-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11233.288774] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [11241.472622] Lustre: server umount fs1259-OST0000 complete [11251.325665] Lustre: server umount fs1259-OST0001 complete [11253.748922] LustreError: 304113:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11253.772700] LustreError: 304113:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 38 previous similar messages [11264.030265] loop1: detected capacity change from 0 to 204800000 [11269.633036] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11269.804876] Lustre: fs1259-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11276.041364] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [11283.180242] loop2: detected capacity change from 0 to 204800000 [11289.513591] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11295.744663] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [11305.598295] Lustre: server umount fs1259-OST0000 complete [11316.078640] Lustre: server umount fs1259-OST0001 complete [11330.536510] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11330.547483] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11330.568281] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11330.573431] Lustre: Skipped 1 previous similar message [11335.560742] Lustre: server umount lustre-OST0000 complete [11340.777504] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11340.807218] Lustre: Skipped 1 previous similar message [11340.817508] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11345.960420] Lustre: server umount lustre-MDT0000 complete [11350.864083] LustreError: 302267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788585898 with bad export cookie 1170052228561897469 [11350.871281] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11351.125223] Lustre: server umount lustre-MDT0001 complete [11365.163360] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [11367.390020] Key type lgssc unregistered [11367.551995] LNet: 315535:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11367.557812] LNetError: 315535:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11367.573996] LNet: Removed LNI 192.168.201.133@tcp [11368.511180] Key type .llcrypt unregistered [11368.514379] Key type ._llcrypt unregistered [11402.900590] Key type ._llcrypt registered [11402.902924] Key type .llcrypt registered [11403.033561] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_hostid [11418.871588] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [11420.388912] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11420.471919] alg: No test for adler32 (adler32-zlib) [11421.613237] Lustre: Lustre: Build Version: 2.17.58_3_g4a08d7b [11421.990601] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [11423.824160] Key type lgssc registered [11425.368817] Lustre: Echo OBD driver; http://www.lustre.org/ [11466.769603] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11466.797522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11468.194830] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11468.235387] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11468.361060] Lustre: lustre-MDT0000: new disk, initializing [11468.487314] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11468.516563] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11473.468059] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [11486.595916] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11486.797518] Lustre: 319394:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11486.874887] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11486.880647] Lustre: Skipped 1 previous similar message [11487.045173] Lustre: lustre-MDT0001: new disk, initializing [11487.213287] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11487.282912] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11487.312721] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11492.232098] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [11503.103646] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11510.090510] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11517.413306] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11517.776453] Lustre: lustre-OST0000: new disk, initializing [11517.781162] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11517.788339] Lustre: 321364:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11517.929632] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11524.691745] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11524.699250] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11524.808414] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11525.217432] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [11536.015318] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11538.463674] Lustre: Mounted lustre-client - version 2.17.58_3_g4a08d7b [11539.466138] Lustre: Unmounted lustre-client [11550.690258] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11550.703918] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11550.714370] LustreError: Skipped 1 previous similar message [11550.741329] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11550.754438] Lustre: Skipped 1 previous similar message [11553.763598] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11553.777790] Lustre: Skipped 1 previous similar message [11555.677548] Lustre: server umount lustre-OST0000 complete [11564.008705] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11564.010845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11564.019946] Lustre: Skipped 2 previous similar messages [11564.020929] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11564.051296] Lustre: Skipped 1 previous similar message [11566.663844] Lustre: server umount lustre-MDT0000 complete [11571.101165] LustreError: 319386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788586118 with bad export cookie 14740258286466255076 [11571.106571] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11571.123350] LustreError: 319386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11571.835228] Lustre: server umount lustre-MDT0001 complete [11584.548677] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [11587.206858] Key type lgssc unregistered [11591.072229] LNet: 1 local NIs in recovery (showing 1): 192.168.201.133@tcp [11591.585354] LNet: 323524:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11591.596817] LNetError: 323524:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11670.520564] LNet: Removed LNI 192.168.201.133@tcp [11671.806786] Key type .llcrypt unregistered [11671.808728] Key type ._llcrypt unregistered [11693.547249] Key type ._llcrypt registered [11693.551402] Key type .llcrypt registered [11693.702423] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_hostid [11711.598105] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [11713.390635] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11713.415274] alg: No test for adler32 (adler32-zlib) [11714.527511] Lustre: Lustre: Build Version: 2.17.58_3_g4a08d7b [11714.863573] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [11716.520192] Key type lgssc registered [11717.793424] Lustre: Echo OBD driver; http://www.lustre.org/ [11754.048756] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 01:31:39 (1788586299) [11762.438840] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11762.505322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11763.789372] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11763.845766] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11764.021338] Lustre: lustre-MDT0000: new disk, initializing [11764.157412] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11764.198461] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11768.881894] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [11782.261379] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11782.489598] Lustre: 326746:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11782.590892] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11782.597423] Lustre: Skipped 1 previous similar message [11782.739628] Lustre: lustre-MDT0001: new disk, initializing [11782.868306] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11782.953834] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11782.962325] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11788.538206] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [11798.869451] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11803.550490] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11809.678523] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11809.904651] Lustre: lustre-OST0000: new disk, initializing [11809.909654] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11809.943829] Lustre: 328713:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11810.053536] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11817.140727] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [11818.601727] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11818.619761] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11818.733850] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11828.094469] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11849.699989] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11849.725764] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11849.777981] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11849.791783] Lustre: Skipped 1 previous similar message [11851.123173] Lustre: server umount lustre-OST0000 complete [11859.937840] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11859.938851] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11859.948094] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11859.949189] LustreError: Skipped 1 previous similar message [11860.003104] Lustre: Skipped 2 previous similar messages [11861.841778] Lustre: server umount lustre-MDT0000 complete [11866.251413] LustreError: 326739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788586413 with bad export cookie 11897560601215175994 [11866.264726] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11866.268141] LustreError: 326739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11866.862234] Lustre: server umount lustre-MDT0001 complete [11880.508726] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [11883.804698] Key type lgssc unregistered [11884.089474] LNet: 330622:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11884.099288] LNetError: 330622:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11884.120559] LNet: Removed LNI 192.168.201.133@tcp [11885.275975] Key type .llcrypt unregistered [11885.278487] Key type ._llcrypt unregistered [11909.324204] Key type ._llcrypt registered [11909.329223] Key type .llcrypt registered [11909.469111] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [11911.410633] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11911.452610] alg: No test for adler32 (adler32-zlib) [11912.544307] Lustre: Lustre: Build Version: 2.17.58_3_g4a08d7b [11912.871082] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [11914.584176] Key type lgssc registered [11915.941864] Lustre: Echo OBD driver; http://www.lustre.org/ [11924.589647] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11924.615599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11926.196979] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11931.626557] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [11935.714333] LustreError: 331892:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11935.743477] LustreError: 331892:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11940.834083] LustreError: 331893:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11942.217945] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11942.890918] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11948.084642] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [11956.803193] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11963.816591] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11971.278076] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11971.907393] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11979.122467] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [11986.864271] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12008.293135] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12009.802728] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12014.569963] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12014.578346] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12014.595863] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.33@tcp (stopping) [12014.606575] Lustre: Skipped 1 previous similar message [12016.110272] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12016.112963] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12016.159586] Lustre: Skipped 2 previous similar messages [12019.668817] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.33@tcp (stopping) [12024.794495] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.33@tcp (stopping) [12024.803711] Lustre: Skipped 3 previous similar messages [12028.384186] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12028.613156] Lustre: server umount lustre-MDT0000 complete [12029.921474] LustreError: 334384:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12032.303708] LustreError: 331873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788586579 with bad export cookie 3140524908638059756 [12032.304979] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12032.322990] LustreError: 331873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12035.037394] LustreError: 331889:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12035.064703] LustreError: 331889:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12035.068691] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.33@tcp (stopping) [12035.102872] Lustre: Skipped 3 previous similar messages [12036.579638] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12036.596645] Lustre: Skipped 1 previous similar message [12038.947277] Lustre: server umount lustre-MDT0001 complete [12042.933175] Lustre: server umount lustre-OST0000 complete [12052.635226] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12058.015080] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [12061.280524] Lustre: 336107:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.133@tcp [12063.967388] Lustre: server umount MGS complete [12074.128569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12074.474067] LustreError: 336679:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12074.489512] LustreError: 336679:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12074.546945] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12080.024177] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [12084.706504] LustreError: 336679:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12084.726513] LustreError: 336679:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12087.878229] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12088.234623] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12091.725706] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [12099.800506] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12100.238988] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12107.980636] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [12117.526282] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12120.828562] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12128.983452] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12131.016765] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12138.465960] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12138.473326] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12138.503516] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12138.507373] Lustre: Skipped 2 previous similar messages [12140.513349] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12143.939378] Lustre: server umount lustre-OST0000 complete [12149.729458] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12149.739345] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12149.755420] Lustre: Skipped 1 previous similar message [12154.440422] Lustre: server umount lustre-MDT0000 complete [12155.891218] LustreError: 336675:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12155.913881] LustreError: 336675:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12158.849557] LustreError: 336660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788586706 with bad export cookie 3140524908638060603 [12158.855500] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12158.867285] LustreError: 336660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12159.182797] Lustre: server umount lustre-MDT0001 complete [12171.397870] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [12174.231708] Key type lgssc unregistered [12174.558848] LNet: 340114:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12174.566464] LNetError: 340114:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12174.599182] LNet: Removed LNI 192.168.201.133@tcp [12175.558181] Key type .llcrypt unregistered [12175.559858] Key type ._llcrypt unregistered [12196.212894] Key type ._llcrypt registered [12196.215665] Key type .llcrypt registered [12196.336040] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_hostid [12214.080642] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [12215.281761] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12215.465181] alg: No test for adler32 (adler32-zlib) [12216.758899] Lustre: Lustre: Build Version: 2.17.58_3_g4a08d7b [12217.115749] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [12218.856316] Key type lgssc registered [12220.271964] Lustre: Echo OBD driver; http://www.lustre.org/ [12255.139436] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12255.190895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12256.647925] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12256.772467] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12256.918672] Lustre: lustre-MDT0000: new disk, initializing [12257.033392] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12257.061034] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12261.713268] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [12273.996640] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12274.261185] Lustre: 343273:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12274.413337] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12274.426806] Lustre: Skipped 1 previous similar message [12274.680713] Lustre: lustre-MDT0001: new disk, initializing [12274.940513] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12275.011436] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12275.040979] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12280.804356] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [12291.669594] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12297.958627] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12304.962348] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12305.294360] Lustre: lustre-OST0000: new disk, initializing [12305.299242] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12305.309415] Lustre: 345241:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12305.429350] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12311.434462] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [12314.239309] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12314.254817] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12314.493085] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12319.805973] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12324.897095] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12325.095504] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12331.134343] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12331.461088] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12335.076108] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12335.085068] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12335.108160] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12335.115069] Lustre: Skipped 1 previous similar message [12335.140756] Lustre: Skipped 1 previous similar message [12339.772861] Lustre: server umount lustre-OST0000 complete [12345.319812] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12345.321139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12345.332099] Lustre: Skipped 1 previous similar message [12349.719853] Lustre: server umount lustre-MDT0000 complete [12353.481413] LustreError: 343266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788586901 with bad export cookie 8760086815905123023 [12353.488336] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12353.494593] LustreError: 343266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12353.907208] Lustre: server umount lustre-MDT0001 complete [12363.158672] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 01:41:48 (1788586908) [12381.474870] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_hostid [12392.360589] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [12431.668350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12431.888422] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12431.926870] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12432.014883] Lustre: lustre-MDT0000: new disk, initializing [12432.126419] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12432.153203] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12436.369864] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [12446.781413] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12446.877477] Lustre: 350002:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12446.920276] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12446.922684] Lustre: Skipped 1 previous similar message [12446.972568] Lustre: lustre-MDT0001: new disk, initializing [12447.038081] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12447.068531] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12447.074900] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12450.301995] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [12459.965517] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12465.449339] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12472.672755] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12473.016718] Lustre: lustre-OST0000: new disk, initializing [12473.025926] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12473.042284] Lustre: 351972:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12473.158387] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12480.335281] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [12481.680460] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12481.701183] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12481.882237] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12491.608911] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12528.100494] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12528.113489] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12528.150683] LustreError: Skipped 1 previous similar message [12528.155253] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12528.181622] Lustre: Skipped 2 previous similar messages [12533.226503] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12533.233811] Lustre: Skipped 1 previous similar message [12538.337430] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12538.356968] Lustre: Skipped 1 previous similar message [12539.360146] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12539.637296] Lustre: server umount lustre-OST0000 complete [12544.481763] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12544.487066] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12544.495662] Lustre: Skipped 1 previous similar message [12548.581473] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12548.583119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12548.611688] Lustre: Skipped 2 previous similar messages [12558.304160] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12558.699386] Lustre: server umount lustre-MDT0000 complete [12558.818277] LustreError: 351324:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12558.850730] LustreError: 351324:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12563.602093] LustreError: 349993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788587111 with bad export cookie 8760086815905123905 [12563.605135] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12563.613326] LustreError: 349993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12563.926905] Lustre: server umount lustre-MDT0001 complete [12576.385759] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [12580.317420] Key type lgssc unregistered [12580.665877] LNet: 354151:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12580.675270] LNetError: 354151:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12581.734419] LNet: Removed LNI 192.168.201.133@tcp [12582.911236] Key type .llcrypt unregistered [12582.914190] Key type ._llcrypt unregistered [12677.715457] Key type ._llcrypt registered [12677.718461] Key type .llcrypt registered [12677.815129] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [12678.093391] Key type .llcrypt unregistered [12678.096123] Key type ._llcrypt unregistered [12753.672253] Key type ._llcrypt registered [12753.673827] Key type .llcrypt registered [12753.773581] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [12753.965494] Key type .llcrypt unregistered [12753.967239] Key type ._llcrypt unregistered [12784.746119] Key type ._llcrypt registered [12784.747960] Key type .llcrypt registered [12784.863828] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [12785.196424] Key type .llcrypt unregistered [12785.200773] Key type ._llcrypt unregistered [12813.820249] Key type ._llcrypt registered [12813.829586] Key type .llcrypt registered [12813.961276] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [12814.349602] Key type .llcrypt unregistered [12814.353550] Key type ._llcrypt unregistered [12903.505128] Key type ._llcrypt registered [12903.511269] Key type .llcrypt registered [12903.614298] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [12903.871289] Key type .llcrypt unregistered [12903.873100] Key type ._llcrypt unregistered [12942.606584] Key type ._llcrypt registered [12942.608818] Key type .llcrypt registered [12942.728379] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [12943.270359] Key type .llcrypt unregistered [12943.278234] Key type ._llcrypt unregistered [12983.132551] Key type ._llcrypt registered [12983.155569] Key type .llcrypt registered [12983.343595] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [12983.866688] Key type .llcrypt unregistered [12983.868417] Key type ._llcrypt unregistered [13023.602822] Key type ._llcrypt registered [13023.608866] Key type .llcrypt registered [13023.758414] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [13024.113326] Key type .llcrypt unregistered [13024.115986] Key type ._llcrypt unregistered [13062.823695] Key type ._llcrypt registered [13062.826749] Key type .llcrypt registered [13062.963990] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [13063.445536] Key type .llcrypt unregistered [13063.449978] Key type ._llcrypt unregistered [13101.219337] Key type ._llcrypt registered [13101.222422] Key type .llcrypt registered [13101.374349] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [13101.888987] Key type .llcrypt unregistered [13101.894230] Key type ._llcrypt unregistered [13177.928538] Key type ._llcrypt registered [13177.932528] Key type .llcrypt registered [13178.161882] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [13178.640362] Key type .llcrypt unregistered [13178.643193] Key type ._llcrypt unregistered [13217.565776] Key type ._llcrypt registered [13217.569304] Key type .llcrypt registered [13217.706144] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [13218.205621] Key type .llcrypt unregistered [13218.209088] Key type ._llcrypt unregistered [13256.373711] Key type ._llcrypt registered [13256.378875] Key type .llcrypt registered [13256.532698] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [13256.922465] Key type .llcrypt unregistered [13256.927075] Key type ._llcrypt unregistered [13322.595419] Key type ._llcrypt registered [13322.596930] Key type .llcrypt registered [13322.784336] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_hostid [13343.496455] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [13345.130767] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13345.175047] alg: No test for adler32 (adler32-zlib) [13346.330658] Lustre: Lustre: Build Version: 2.17.58_3_g4a08d7b [13346.627604] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [13348.368248] Key type lgssc registered [13350.115517] Lustre: Echo OBD driver; http://www.lustre.org/ [13395.765163] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13395.795717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13397.182974] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [13397.232837] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [13397.353797] Lustre: tst250fs-MDT0000: new disk, initializing [13397.484494] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13397.507928] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13403.054404] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [13416.241257] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13416.355443] Lustre: 374447:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13416.405165] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13416.407228] Lustre: Skipped 1 previous similar message [13416.449262] Lustre: tst250fs-MDT0001: new disk, initializing [13416.565409] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13416.600809] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13416.612708] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13422.920956] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [13428.984931] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13437.393510] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13437.845665] Lustre: tst250fs-OST0000: new disk, initializing [13437.854271] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13437.878639] Lustre: 376079:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13438.162907] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13438.867772] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13438.880579] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13439.044826] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13447.040574] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [13462.422944] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13462.687949] Lustre: tst250fs-OST0001: new disk, initializing [13462.696162] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13462.709487] Lustre: 376954:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13462.885775] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13470.734750] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [13471.860348] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13471.881089] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13472.044574] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13477.078976] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13479.166454] Lustre: Setting parameter tst250fs.llite.tst250fs-*.max_read_ahead_mb=32 in log params [13479.172108] Lustre: Skipped 1 previous similar message [13502.974299] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13502.990072] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13502.996653] Lustre: Skipped 1 previous similar message [13503.023598] Lustre: Skipped 3 previous similar messages [13508.004908] Lustre: server umount tst250fs-MDT0000 complete [13513.194200] LustreError: 374459:0:(ldlm_lib.c:1190:target_handle_connect()) tst250fs-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13513.236335] LustreError: 374459:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [13514.379444] LustreError: 376078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788588062 with bad export cookie 13858152668142632293 [13514.381257] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13514.391985] LustreError: 376078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13514.741083] Lustre: server umount tst250fs-MDT0001 complete [13529.948044] Lustre: server umount tst250fs-OST0000 complete [13533.859303] Lustre: 372672:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788588065/real 1788588065] req@ffff9851cefcfb80 x1875470339616128/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788588081 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13533.925216] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13533.945052] Lustre: Skipped 2 previous similar messages [13535.764528] Lustre: server umount tst250fs-OST0001 complete [13559.344767] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_hostid [13573.412427] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [13619.199870] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 02:02:43 (1788588163) [13621.637780] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13647.979206] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_hostid [13663.573754] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [13703.085778] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13306 sec ========= 02:04:08 (1788588248) [13705.286679] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 02:04:10 (1788588250) === [13710.430535] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 02:04:15 (1788588255) === [13763.031378] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [13767.468469] Key type lgssc unregistered [13767.833755] LNet: 386135:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13767.839607] LNetError: 386135:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13768.883968] LNet: Removed LNI 192.168.201.133@tcp [13770.110596] Key type .llcrypt unregistered [13770.113315] Key type ._llcrypt unregistered